[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:01:52 dist-test-slave-k7z7 krb5kdc[1662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:01:52 dist-test-slave-k7z7 krb5kdc[1662](info): set up 2 sockets
Dec 14 03:01:52 dist-test-slave-k7z7 krb5kdc[1662](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251214 03:01:54.697588 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.814s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:01:54 dist-test-slave-k7z7 krb5kdc[1662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:01:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:01:54.833675 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:42395
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:01:54.939841 1678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:01:54.940117 1678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:01:54.940197 1678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:01:54.943619 1678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:01:54.943691 1678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:01:54.943722 1678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:01:54.943745 1678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:01:54.943773 1678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:01:54.948069 1678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:42395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:01:54.949137 1678 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:01:54.950018 1678 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:01:54.956475 1686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:54.956452 1684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:54.956458 1683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:01:54.956553 1678 server_base.cc:1047] running on GCE node
I20251214 03:01:54.957298 1678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:01:54.958345 1678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:01:54.959519 1678 hybrid_clock.cc:648] HybridClock initialized: now 1765681314959470 us; error 67 us; skew 500 ppm
Dec 14 03:01:54 dist-test-slave-k7z7 krb5kdc[1662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:01:54.962075 1678 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:01:54.963131 1678 webserver.cc:492] Webserver started at http://127.1.157.254:44353/ using document root <none> and password file <none>
I20251214 03:01:54.963629 1678 fs_manager.cc:362] Metadata directory not provided
I20251214 03:01:54.963673 1678 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:01:54.963821 1678 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:01:54.965545 1678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20"
format_stamp: "Formatted at 2025-12-14 03:01:54 on dist-test-slave-k7z7"
server_key: "fc11790e691ff2cbb263ea3871d868ff"
server_key_iv: "93962fde03ae658863cb76d200553cf0"
server_key_version: "encryptionkey@0"
I20251214 03:01:54.965969 1678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20"
format_stamp: "Formatted at 2025-12-14 03:01:54 on dist-test-slave-k7z7"
server_key: "fc11790e691ff2cbb263ea3871d868ff"
server_key_iv: "93962fde03ae658863cb76d200553cf0"
server_key_version: "encryptionkey@0"
I20251214 03:01:54.969204 1678 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:01:54.971385 1693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:54.972368 1678 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:01:54.972487 1678 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20"
format_stamp: "Formatted at 2025-12-14 03:01:54 on dist-test-slave-k7z7"
server_key: "fc11790e691ff2cbb263ea3871d868ff"
server_key_iv: "93962fde03ae658863cb76d200553cf0"
server_key_version: "encryptionkey@0"
I20251214 03:01:54.972580 1678 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:01:54.983669 1678 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:01:54.990909 1678 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:01:54.991112 1678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:01:54.998010 1678 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:42395
I20251214 03:01:54.998011 1745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:42395 every 8 connection(s)
I20251214 03:01:54.999027 1678 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:01:55.003822 1746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:01:55.009150 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 1678
I20251214 03:01:55.009346 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:01:55.010341 1655 external_mini_cluster.cc:1468] Setting key d63b53244335d8e19849c0125bf242d5
I20251214 03:01:55.010622 1746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20: Bootstrap starting.
I20251214 03:01:55.013296 1746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20: Neither blocks nor log segments found. Creating new log.
I20251214 03:01:55.014268 1746 log.cc:826] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20: Log is configured to *not* fsync() on all Append() calls
I20251214 03:01:55.016423 1746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20: No bootstrap required, opened a new log
I20251214 03:01:55.019505 1746 raft_consensus.cc:359] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42395 } }
I20251214 03:01:55.019677 1746 raft_consensus.cc:385] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:01:55.019721 1746 raft_consensus.cc:740] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311bce0f7ffe4ddba4f582ff1f3b5a20, State: Initialized, Role: FOLLOWER
I20251214 03:01:55.020135 1746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42395 } }
I20251214 03:01:55.020284 1746 raft_consensus.cc:399] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:01:55.020342 1746 raft_consensus.cc:493] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:01:55.020424 1746 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:01:55.021536 1746 raft_consensus.cc:515] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42395 } }
I20251214 03:01:55.021884 1746 leader_election.cc:304] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 311bce0f7ffe4ddba4f582ff1f3b5a20; no voters:
I20251214 03:01:55.022166 1746 leader_election.cc:290] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:01:55.022271 1751 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:01:55.022423 1751 raft_consensus.cc:697] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [term 1 LEADER]: Becoming Leader. State: Replica: 311bce0f7ffe4ddba4f582ff1f3b5a20, State: Running, Role: LEADER
I20251214 03:01:55.022760 1751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42395 } }
I20251214 03:01:55.023288 1746 sys_catalog.cc:565] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [sys.catalog]: configured and running, proceeding with master startup.
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:01:55.023874 1753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 311bce0f7ffe4ddba4f582ff1f3b5a20. Latest consensus state: current_term: 1 leader_uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42395 } } }
I20251214 03:01:55.023964 1753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [sys.catalog]: This master's current role is: LEADER
I20251214 03:01:55.024859 1752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311bce0f7ffe4ddba4f582ff1f3b5a20" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42395 } } }
I20251214 03:01:55.024941 1752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20 [sys.catalog]: This master's current role is: LEADER
I20251214 03:01:55.025518 1760 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:01:55.028326 1760 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:01:55.030050 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:55.012719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.1.157.254:42395)
1214 03:01:55.013275 (+ 556us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:55.013287 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:55.013717 (+ 430us) server_negotiation.cc:408] Connection header received
1214 03:01:55.016462 (+ 2745us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:55.016484 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:55.016776 (+ 292us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:55.017030 (+ 254us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:55.019197 (+ 2167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.020025 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:55.020851 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.021051 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:55.024712 (+ 3661us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:55.024735 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:55.024747 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:55.024786 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:55.026959 (+ 2173us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.027525 (+ 566us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.027531 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.027535 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.027620 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.027926 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.027930 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.027932 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.028324 (+ 392us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:55.028530 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:55.029100 (+ 570us) server_negotiation.cc:300] Negotiation successful
1214 03:01:55.029364 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":123,"threads_started":1}
I20251214 03:01:55.033613 1760 catalog_manager.cc:1357] Generated new cluster ID: d1202d08b35d4aef8ec3ec7d855bdee5
I20251214 03:01:55.033682 1760 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:01:55.043243 1760 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:01:55.044544 1760 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:01:55.058050 1760 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 311bce0f7ffe4ddba4f582ff1f3b5a20: Generated new TSK 0
I20251214 03:01:55.058936 1760 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:01:55.103785 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42395
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:01:55.209669 1774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:01:55.209882 1774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:01:55.209945 1774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:01:55.213153 1774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:01:55.213244 1774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:01:55.213341 1774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:01:55.217520 1774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:42395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:01:55.218652 1774 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:01:55.219502 1774 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:01:55.226019 1780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:55.226073 1782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:01:55.226328 1774 server_base.cc:1047] running on GCE node
W20251214 03:01:55.226022 1779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:01:55.226819 1774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:01:55.227355 1774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:01:55.228539 1774 hybrid_clock.cc:648] HybridClock initialized: now 1765681315228520 us; error 35 us; skew 500 ppm
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:01:55.231165 1774 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:01:55.232200 1774 webserver.cc:492] Webserver started at http://127.1.157.193:40265/ using document root <none> and password file <none>
I20251214 03:01:55.232718 1774 fs_manager.cc:362] Metadata directory not provided
I20251214 03:01:55.232767 1774 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:01:55.232932 1774 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:01:55.234666 1774 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "024020985cb54411a27dc19a8d75eeb8"
format_stamp: "Formatted at 2025-12-14 03:01:55 on dist-test-slave-k7z7"
server_key: "8c251ecca24e903fc7aaf6576952ab63"
server_key_iv: "12ea26f2af5499b1c334709e1fb7a1cc"
server_key_version: "encryptionkey@0"
I20251214 03:01:55.235136 1774 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "024020985cb54411a27dc19a8d75eeb8"
format_stamp: "Formatted at 2025-12-14 03:01:55 on dist-test-slave-k7z7"
server_key: "8c251ecca24e903fc7aaf6576952ab63"
server_key_iv: "12ea26f2af5499b1c334709e1fb7a1cc"
server_key_version: "encryptionkey@0"
I20251214 03:01:55.238476 1774 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:01:55.240603 1789 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:55.241662 1774 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:01:55.241766 1774 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "024020985cb54411a27dc19a8d75eeb8"
format_stamp: "Formatted at 2025-12-14 03:01:55 on dist-test-slave-k7z7"
server_key: "8c251ecca24e903fc7aaf6576952ab63"
server_key_iv: "12ea26f2af5499b1c334709e1fb7a1cc"
server_key_version: "encryptionkey@0"
I20251214 03:01:55.241853 1774 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:01:55.251363 1774 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:01:55.253945 1774 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:01:55.254118 1774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:01:55.254654 1774 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:01:55.255543 1774 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:01:55.255595 1774 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:55.255646 1774 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:01:55.255674 1774 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:55.264323 1774 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:36325
I20251214 03:01:55.264356 1902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:36325 every 8 connection(s)
I20251214 03:01:55.265448 1774 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:01:55.268822 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 1774
I20251214 03:01:55.268926 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:01:55.269198 1655 external_mini_cluster.cc:1468] Setting key a60f34e68864ba15ed80dc7d43788149
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:01:55.280495 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:55.268032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:47149 (local address 127.1.157.254:42395)
1214 03:01:55.268221 (+ 189us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:55.268225 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:55.268927 (+ 702us) server_negotiation.cc:408] Connection header received
1214 03:01:55.269923 (+ 996us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:55.269927 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:55.269987 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:55.270060 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:55.271965 (+ 1905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.272657 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:55.273412 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.273610 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:55.275710 (+ 2100us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:55.275739 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:55.275743 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:55.275776 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:55.277961 (+ 2185us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.278698 (+ 737us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.278703 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.278704 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.278757 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.279247 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.279251 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.279252 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.279453 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:55.279548 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:55.280153 (+ 605us) server_negotiation.cc:300] Negotiation successful
1214 03:01:55.280315 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
I20251214 03:01:55.281147 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:55.268303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42395 (local address 127.1.157.193:47149)
1214 03:01:55.268770 (+ 467us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:55.268810 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:55.269723 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:55.270264 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:55.270276 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:55.270835 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:01:55.271814 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:55.271830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.272808 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:55.272812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:55.273288 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:55.273296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.273490 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:55.274054 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:01:55.274079 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:01:55.275536 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:01:55.278132 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:55.278139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:55.278152 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:55.278437 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:55.278919 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:55.278923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:55.278925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:55.279078 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:55.279596 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:01:55.279602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:01:55.279871 (+ 269us) client_negotiation.cc:769] Sending connection context
1214 03:01:55.280076 (+ 205us) client_negotiation.cc:240] Negotiation successful
1214 03:01:55.280357 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":145,"threads_started":1}
I20251214 03:01:55.282526 1903 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42395
I20251214 03:01:55.282791 1903 heartbeater.cc:461] Registering TS with master...
I20251214 03:01:55.283334 1903 heartbeater.cc:507] Master 127.1.157.254:42395 requested a full tablet report, sending...
I20251214 03:01:55.284817 1710 ts_manager.cc:194] Registered new tserver with Master: 024020985cb54411a27dc19a8d75eeb8 (127.1.157.193:36325)
I20251214 03:01:55.286247 1710 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:47149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:01:55.320817 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42395
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:01:55.434501 1910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:01:55.434733 1910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:01:55.434813 1910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:01:55.438119 1910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:01:55.438190 1910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:01:55.438298 1910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:01:55.442685 1910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:42395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:01:55.444124 1910 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:01:55.445219 1910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:01:55.453032 1915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:55.453039 1918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:01:55.453243 1910 server_base.cc:1047] running on GCE node
W20251214 03:01:55.453122 1916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:01:55.453713 1910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:01:55.454267 1910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:01:55.455435 1910 hybrid_clock.cc:648] HybridClock initialized: now 1765681315455412 us; error 37 us; skew 500 ppm
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:01:55.458060 1910 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:01:55.459065 1910 webserver.cc:492] Webserver started at http://127.1.157.194:41501/ using document root <none> and password file <none>
I20251214 03:01:55.459592 1910 fs_manager.cc:362] Metadata directory not provided
I20251214 03:01:55.459640 1910 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:01:55.459805 1910 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:01:55.461560 1910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "e686442e8d38420e953bd8a87d45ecf2"
format_stamp: "Formatted at 2025-12-14 03:01:55 on dist-test-slave-k7z7"
server_key: "0d48ad16469c13f7909f6c4c02661cee"
server_key_iv: "4ec933e195929b93dba5d182b28f4bd6"
server_key_version: "encryptionkey@0"
I20251214 03:01:55.462005 1910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "e686442e8d38420e953bd8a87d45ecf2"
format_stamp: "Formatted at 2025-12-14 03:01:55 on dist-test-slave-k7z7"
server_key: "0d48ad16469c13f7909f6c4c02661cee"
server_key_iv: "4ec933e195929b93dba5d182b28f4bd6"
server_key_version: "encryptionkey@0"
I20251214 03:01:55.465598 1910 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:01:55.467900 1925 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:55.468952 1910 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:01:55.469059 1910 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "e686442e8d38420e953bd8a87d45ecf2"
format_stamp: "Formatted at 2025-12-14 03:01:55 on dist-test-slave-k7z7"
server_key: "0d48ad16469c13f7909f6c4c02661cee"
server_key_iv: "4ec933e195929b93dba5d182b28f4bd6"
server_key_version: "encryptionkey@0"
I20251214 03:01:55.469151 1910 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:01:55.513938 1910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:01:55.516731 1910 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:01:55.516901 1910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:01:55.517501 1910 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:01:55.518375 1910 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:01:55.518428 1910 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:55.518481 1910 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:01:55.518508 1910 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:55.527104 1910 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:39295
I20251214 03:01:55.527132 2038 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:39295 every 8 connection(s)
I20251214 03:01:55.528009 1910 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:01:55.537560 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 1910
I20251214 03:01:55.537653 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:01:55.537856 1655 external_mini_cluster.cc:1468] Setting key 2762873c6cb639ddbab54666284c36c4
I20251214 03:01:55.539971 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:55.529631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:39631 (local address 127.1.157.254:42395)
1214 03:01:55.529745 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:55.529749 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:55.530549 (+ 800us) server_negotiation.cc:408] Connection header received
1214 03:01:55.531373 (+ 824us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:55.531376 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:55.531435 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:55.531507 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:55.533106 (+ 1599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.533640 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:55.534259 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.534429 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:55.536507 (+ 2078us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:55.536527 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:55.536529 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:55.536555 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:55.538118 (+ 1563us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.538576 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.538578 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.538580 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.538622 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.538897 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.538899 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.538900 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.539071 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:55.539183 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:55.539700 (+ 517us) server_negotiation.cc:300] Negotiation successful
1214 03:01:55.539822 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251214 03:01:55.540480 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:55.529990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42395 (local address 127.1.157.194:39631)
1214 03:01:55.530418 (+ 428us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:55.530451 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:55.531185 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:55.531670 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:55.531678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:55.532050 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:01:55.532960 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:55.532975 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.533753 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:55.533756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:55.534153 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:55.534159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.534359 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:55.534922 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:01:55.534943 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:01:55.536342 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:01:55.538236 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:55.538241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:55.538250 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:55.538485 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:55.538697 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:55.538700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:55.538702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:55.538823 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:55.539175 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:01:55.539182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:01:55.539430 (+ 248us) client_negotiation.cc:769] Sending connection context
1214 03:01:55.539606 (+ 176us) client_negotiation.cc:240] Negotiation successful
1214 03:01:55.539815 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
I20251214 03:01:55.541543 2039 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42395
I20251214 03:01:55.541769 2039 heartbeater.cc:461] Registering TS with master...
I20251214 03:01:55.542266 2039 heartbeater.cc:507] Master 127.1.157.254:42395 requested a full tablet report, sending...
I20251214 03:01:55.543259 1710 ts_manager.cc:194] Registered new tserver with Master: e686442e8d38420e953bd8a87d45ecf2 (127.1.157.194:39295)
I20251214 03:01:55.543996 1710 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:39631
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:01:55.584270 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42395
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:01:55.697556 2046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:01:55.697775 2046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:01:55.697870 2046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:01:55.701110 2046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:01:55.701211 2046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:01:55.701318 2046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:01:55.705411 2046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:42395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:01:55.706401 2046 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:01:55.707208 2046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:01:55.713807 2051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:55.713891 2052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:55.713894 2054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:01:55.714035 2046 server_base.cc:1047] running on GCE node
I20251214 03:01:55.714416 2046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:01:55.714910 2046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:01:55.716061 2046 hybrid_clock.cc:648] HybridClock initialized: now 1765681315716044 us; error 34 us; skew 500 ppm
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:01:55.718498 2046 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:01:55.719426 2046 webserver.cc:492] Webserver started at http://127.1.157.195:42579/ using document root <none> and password file <none>
I20251214 03:01:55.719895 2046 fs_manager.cc:362] Metadata directory not provided
I20251214 03:01:55.719936 2046 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:01:55.720077 2046 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:01:55.721695 2046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "fca9d4b86a3542a198d80255cadf8832"
format_stamp: "Formatted at 2025-12-14 03:01:55 on dist-test-slave-k7z7"
server_key: "33dbc88ddd1e51f4abb66cec21815435"
server_key_iv: "475791e1830b83a2a87618a5fda69824"
server_key_version: "encryptionkey@0"
I20251214 03:01:55.722102 2046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "fca9d4b86a3542a198d80255cadf8832"
format_stamp: "Formatted at 2025-12-14 03:01:55 on dist-test-slave-k7z7"
server_key: "33dbc88ddd1e51f4abb66cec21815435"
server_key_iv: "475791e1830b83a2a87618a5fda69824"
server_key_version: "encryptionkey@0"
I20251214 03:01:55.725256 2046 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:01:55.727167 2061 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:55.728096 2046 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:01:55.728191 2046 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "fca9d4b86a3542a198d80255cadf8832"
format_stamp: "Formatted at 2025-12-14 03:01:55 on dist-test-slave-k7z7"
server_key: "33dbc88ddd1e51f4abb66cec21815435"
server_key_iv: "475791e1830b83a2a87618a5fda69824"
server_key_version: "encryptionkey@0"
I20251214 03:01:55.728261 2046 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:01:55.754467 2046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:01:55.758047 2046 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:01:55.758296 2046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:01:55.758968 2046 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:01:55.760075 2046 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:01:55.760145 2046 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:55.760200 2046 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:01:55.760269 2046 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251214 03:01:55.769263 2046 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:39349
I20251214 03:01:55.769280 2174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:39349 every 8 connection(s)
I20251214 03:01:55.770159 2046 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:01:55.780160 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 2046
I20251214 03:01:55.780252 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:01:55.780437 1655 external_mini_cluster.cc:1468] Setting key 19f1e2a7f7347bde819c46c60bab7e1f
I20251214 03:01:55.781685 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:55.771748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:47765 (local address 127.1.157.254:42395)
1214 03:01:55.771875 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:55.771879 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:55.772501 (+ 622us) server_negotiation.cc:408] Connection header received
1214 03:01:55.773370 (+ 869us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:55.773374 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:55.773439 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:55.773505 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:55.775061 (+ 1556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.775573 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:55.776204 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.776378 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:55.778577 (+ 2199us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:55.778598 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:55.778601 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:55.778631 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:55.779786 (+ 1155us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.780237 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.780240 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.780242 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.780287 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.780572 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.780576 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.780578 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.780773 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:55.780883 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:55.781403 (+ 520us) server_negotiation.cc:300] Negotiation successful
1214 03:01:55.781534 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251214 03:01:55.782351 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:55.771972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42395 (local address 127.1.157.195:47765)
1214 03:01:55.772377 (+ 405us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:55.772410 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:55.773148 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:55.773662 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:55.773670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:55.774040 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:01:55.774917 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:55.774931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.775681 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:55.775684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:55.776093 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:55.776100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.776303 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:55.776910 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:01:55.776933 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:01:55.778422 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:01:55.779902 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:55.779907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:55.779918 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:55.780124 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:55.780362 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:55.780366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:55.780368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:55.780473 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:55.780871 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:01:55.780877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:01:55.781126 (+ 249us) client_negotiation.cc:769] Sending connection context
1214 03:01:55.781358 (+ 232us) client_negotiation.cc:240] Negotiation successful
1214 03:01:55.781626 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20251214 03:01:55.783257 2175 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42395
I20251214 03:01:55.783470 2175 heartbeater.cc:461] Registering TS with master...
I20251214 03:01:55.783922 2175 heartbeater.cc:507] Master 127.1.157.254:42395 requested a full tablet report, sending...
I20251214 03:01:55.784761 1710 ts_manager.cc:194] Registered new tserver with Master: fca9d4b86a3542a198d80255cadf8832 (127.1.157.195:39349)
I20251214 03:01:55.785308 1710 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:47765
I20251214 03:01:55.794411 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.157.193@KRBTEST.COM
I20251214 03:01:55.817677 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:55.809243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.1.157.193:36325)
1214 03:01:55.809517 (+ 274us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:55.809523 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:55.809549 (+ 26us) server_negotiation.cc:408] Connection header received
1214 03:01:55.809628 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:55.809633 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:55.809805 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:55.809919 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:55.810721 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.811449 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:55.812060 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.812304 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:55.814435 (+ 2131us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:55.814478 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:55.814490 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:55.814521 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:55.816154 (+ 1633us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.816546 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.816551 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.816552 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.816600 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.816817 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.816819 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.816821 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.817006 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:55.817164 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:55.817339 (+ 175us) server_negotiation.cc:300] Negotiation successful
1214 03:01:55.817491 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
W20251214 03:01:55.818632 1817 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54514
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[1662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.193@KRBTEST.COM
I20251214 03:01:55.843230 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:55.834901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.1.157.193:36325)
1214 03:01:55.835048 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:55.835053 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:55.835326 (+ 273us) server_negotiation.cc:408] Connection header received
1214 03:01:55.835500 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:55.835502 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:55.835561 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:55.835629 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:55.836471 (+ 842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.836958 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:55.837765 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:55.837997 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:55.840169 (+ 2172us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:55.840193 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:55.840195 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:55.840226 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:55.841771 (+ 1545us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.842171 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.842175 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.842176 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.842228 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:55.842438 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:55.842442 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:55.842444 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:55.842598 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:55.842719 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:55.842931 (+ 212us) server_negotiation.cc:300] Negotiation successful
1214 03:01:55.843056 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251214 03:01:55.844339 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 1774
I20251214 03:01:55.850206 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 1910
I20251214 03:01:55.857080 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 2046
I20251214 03:01:55.861968 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 1678
2025-12-14T03:01:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3006 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[2189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[2189](info): set up 2 sockets
Dec 14 03:01:55 dist-test-slave-k7z7 krb5kdc[2189](info): commencing operation
krb5kdc: starting...
W20251214 03:01:57.696813 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.814s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:01:57 dist-test-slave-k7z7 krb5kdc[2189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:01:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:01:57.836681 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:46501
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:01:57.948197 2205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:01:57.948439 2205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:01:57.948518 2205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:01:57.951803 2205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:01:57.951876 2205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:01:57.951906 2205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:01:57.951930 2205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:01:57.951959 2205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:01:57.956259 2205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:46501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:01:57.957370 2205 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:01:57.958184 2205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:01:57.963688 2211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:57.963689 2210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:57.963727 2213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:01:57.963974 2205 server_base.cc:1047] running on GCE node
I20251214 03:01:57.964532 2205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:01:57.965440 2205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:01:57.966584 2205 hybrid_clock.cc:648] HybridClock initialized: now 1765681317966572 us; error 30 us; skew 500 ppm
Dec 14 03:01:57 dist-test-slave-k7z7 krb5kdc[2189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:01:57.968926 2205 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:01:57.970018 2205 webserver.cc:492] Webserver started at http://127.1.157.254:41943/ using document root <none> and password file <none>
I20251214 03:01:57.970536 2205 fs_manager.cc:362] Metadata directory not provided
I20251214 03:01:57.970587 2205 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:01:57.970755 2205 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:01:57.972467 2205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c"
format_stamp: "Formatted at 2025-12-14 03:01:57 on dist-test-slave-k7z7"
server_key: "cf0a2c43faa999c0fef6bc52b0d97242"
server_key_iv: "05de92400fbf9e5dd8bcf3c7cef340ed"
server_key_version: "encryptionkey@0"
I20251214 03:01:57.972911 2205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c"
format_stamp: "Formatted at 2025-12-14 03:01:57 on dist-test-slave-k7z7"
server_key: "cf0a2c43faa999c0fef6bc52b0d97242"
server_key_iv: "05de92400fbf9e5dd8bcf3c7cef340ed"
server_key_version: "encryptionkey@0"
I20251214 03:01:57.976269 2205 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:01:57.978552 2220 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:57.979708 2205 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:01:57.979807 2205 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c"
format_stamp: "Formatted at 2025-12-14 03:01:57 on dist-test-slave-k7z7"
server_key: "cf0a2c43faa999c0fef6bc52b0d97242"
server_key_iv: "05de92400fbf9e5dd8bcf3c7cef340ed"
server_key_version: "encryptionkey@0"
I20251214 03:01:57.979893 2205 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:01:57.994380 2205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:01:57.997139 2205 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:01:57.997340 2205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:01:58.003984 2272 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:46501 every 8 connection(s)
I20251214 03:01:58.003991 2205 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:46501
I20251214 03:01:58.004945 2205 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:01:58.007573 2273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:01:58.012728 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 2205
I20251214 03:01:58.012816 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:01:58.012959 2273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c: Bootstrap starting.
I20251214 03:01:58.013006 1655 external_mini_cluster.cc:1468] Setting key e5200669d083b3ead4dc96789af35868
I20251214 03:01:58.015390 2273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c: Neither blocks nor log segments found. Creating new log.
I20251214 03:01:58.016191 2273 log.cc:826] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c: Log is configured to *not* fsync() on all Append() calls
I20251214 03:01:58.018316 2273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c: No bootstrap required, opened a new log
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:01:58.020900 2273 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46501 } }
I20251214 03:01:58.021142 2273 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:01:58.021232 2273 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfcbd2e2e8e4d9891694a0d869faf5c, State: Initialized, Role: FOLLOWER
I20251214 03:01:58.021679 2273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46501 } }
I20251214 03:01:58.021842 2273 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:01:58.021914 2273 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:01:58.022007 2273 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:01:58.022799 2273 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46501 } }
I20251214 03:01:58.023109 2273 leader_election.cc:304] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bfcbd2e2e8e4d9891694a0d869faf5c; no voters:
I20251214 03:01:58.023396 2273 leader_election.cc:290] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:01:58.023761 2278 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:01:58.023993 2278 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [term 1 LEADER]: Becoming Leader. State: Replica: 1bfcbd2e2e8e4d9891694a0d869faf5c, State: Running, Role: LEADER
I20251214 03:01:58.024287 2278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46501 } }
I20251214 03:01:58.024554 2273 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:01:58.025765 2280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bfcbd2e2e8e4d9891694a0d869faf5c. Latest consensus state: current_term: 1 leader_uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46501 } } }
I20251214 03:01:58.025871 2280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [sys.catalog]: This master's current role is: LEADER
I20251214 03:01:58.026090 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.014364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.1.157.254:46501)
1214 03:01:58.014788 (+ 424us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.014800 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.014838 (+ 38us) server_negotiation.cc:408] Connection header received
1214 03:01:58.015505 (+ 667us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.015531 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.015898 (+ 367us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.016237 (+ 339us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:58.017098 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.018227 (+ 1129us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.018894 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.019168 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.020968 (+ 1800us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:58.020989 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:58.021002 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:58.021041 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:58.022861 (+ 1820us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.023276 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.023281 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.023285 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.023359 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.023676 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.023680 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.023683 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.024143 (+ 460us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:58.024350 (+ 207us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.025015 (+ 665us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.025303 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":138,"threads_started":1}
I20251214 03:01:58.026094 2279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfcbd2e2e8e4d9891694a0d869faf5c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46501 } } }
I20251214 03:01:58.026206 2279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c [sys.catalog]: This master's current role is: LEADER
W20251214 03:01:58.029310 2293 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251214 03:01:58.029395 2293 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251214 03:01:58.029453 2287 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:01:58.030318 2287 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:01:58.035110 2287 catalog_manager.cc:1357] Generated new cluster ID: a5976aae8c5b4131a8ed664e7f50fc48
I20251214 03:01:58.035176 2287 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:01:58.050144 2287 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:01:58.051349 2287 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:01:58.058053 2287 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1bfcbd2e2e8e4d9891694a0d869faf5c: Generated new TSK 0
I20251214 03:01:58.058877 2287 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:01:58.108265 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:46501
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:01:58.211503 2301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:01:58.211740 2301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:01:58.211807 2301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:01:58.215211 2301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:01:58.215276 2301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:01:58.215370 2301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:01:58.219616 2301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:46501
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:01:58.220629 2301 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:01:58.221482 2301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:01:58.227794 2309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:58.227877 2307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:58.227794 2306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:01:58.228250 2301 server_base.cc:1047] running on GCE node
I20251214 03:01:58.228587 2301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:01:58.229074 2301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:01:58.230213 2301 hybrid_clock.cc:648] HybridClock initialized: now 1765681318230201 us; error 31 us; skew 500 ppm
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:01:58.232707 2301 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:01:58.233779 2301 webserver.cc:492] Webserver started at http://127.1.157.193:33735/ using document root <none> and password file <none>
I20251214 03:01:58.234293 2301 fs_manager.cc:362] Metadata directory not provided
I20251214 03:01:58.234341 2301 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:01:58.234517 2301 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:01:58.236109 2301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "ff21570316434a33b8d2d0ccd68b374b"
format_stamp: "Formatted at 2025-12-14 03:01:58 on dist-test-slave-k7z7"
server_key: "abdd66d9b53a707adf93aafea61d8ec4"
server_key_iv: "29da252944edf01eb5ec2acc5b2a03af"
server_key_version: "encryptionkey@0"
I20251214 03:01:58.236549 2301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "ff21570316434a33b8d2d0ccd68b374b"
format_stamp: "Formatted at 2025-12-14 03:01:58 on dist-test-slave-k7z7"
server_key: "abdd66d9b53a707adf93aafea61d8ec4"
server_key_iv: "29da252944edf01eb5ec2acc5b2a03af"
server_key_version: "encryptionkey@0"
I20251214 03:01:58.239722 2301 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:01:58.241839 2316 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:58.242820 2301 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:01:58.242919 2301 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "ff21570316434a33b8d2d0ccd68b374b"
format_stamp: "Formatted at 2025-12-14 03:01:58 on dist-test-slave-k7z7"
server_key: "abdd66d9b53a707adf93aafea61d8ec4"
server_key_iv: "29da252944edf01eb5ec2acc5b2a03af"
server_key_version: "encryptionkey@0"
I20251214 03:01:58.243002 2301 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:01:58.260368 2301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:01:58.263350 2301 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:01:58.263633 2301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:01:58.264285 2301 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:01:58.265213 2301 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:01:58.265271 2301 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:58.265326 2301 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:01:58.265349 2301 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:58.274366 2301 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:35923
I20251214 03:01:58.274459 2429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:35923 every 8 connection(s)
I20251214 03:01:58.275338 2301 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:01:58.283604 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 2301
I20251214 03:01:58.283702 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:01:58.283936 1655 external_mini_cluster.cc:1468] Setting key 81f74cf39f105a50f5b980d48c37a4ee
I20251214 03:01:58.287611 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.277223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:51815 (local address 127.1.157.254:46501)
1214 03:01:58.277364 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.277368 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.278028 (+ 660us) server_negotiation.cc:408] Connection header received
1214 03:01:58.278860 (+ 832us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.278863 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.278923 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.278997 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:58.280616 (+ 1619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.281119 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.281770 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.281949 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.284118 (+ 2169us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:58.284143 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:58.284147 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:58.284180 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:58.285801 (+ 1621us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.286235 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.286246 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.286248 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.286296 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.286533 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.286536 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.286538 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.286715 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:58.286809 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.287380 (+ 571us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.287486 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251214 03:01:58.288431 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.277429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:46501 (local address 127.1.157.193:51815)
1214 03:01:58.277877 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:58.277914 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:58.278662 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:58.279161 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:58.279169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:58.279563 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:01:58.280453 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.280476 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.281250 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.281253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:58.281659 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.281667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.281873 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.282449 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:01:58.282485 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:01:58.283969 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:01:58.285912 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:58.285916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:58.285926 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:58.286149 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:58.286374 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:58.286376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:58.286378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:58.286459 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:58.286818 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:01:58.286825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:01:58.287129 (+ 304us) client_negotiation.cc:769] Sending connection context
1214 03:01:58.287361 (+ 232us) client_negotiation.cc:240] Negotiation successful
1214 03:01:58.287649 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":99,"threads_started":1}
I20251214 03:01:58.289600 2430 heartbeater.cc:344] Connected to a master server at 127.1.157.254:46501
I20251214 03:01:58.289824 2430 heartbeater.cc:461] Registering TS with master...
I20251214 03:01:58.290302 2430 heartbeater.cc:507] Master 127.1.157.254:46501 requested a full tablet report, sending...
I20251214 03:01:58.291677 2231 ts_manager.cc:194] Registered new tserver with Master: ff21570316434a33b8d2d0ccd68b374b (127.1.157.193:35923)
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:01:58.292781 2231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:51815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:01:58.328827 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:46501
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:01:58.430925 2437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:01:58.431125 2437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:01:58.431180 2437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:01:58.434293 2437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:01:58.434341 2437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:01:58.434417 2437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:01:58.438370 2437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:46501
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:01:58.439231 2437 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:01:58.439961 2437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:01:58.445820 2442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:58.445816 2443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:58.445850 2445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:01:58.446310 2437 server_base.cc:1047] running on GCE node
I20251214 03:01:58.446619 2437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:01:58.447113 2437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:01:58.448280 2437 hybrid_clock.cc:648] HybridClock initialized: now 1765681318448264 us; error 32 us; skew 500 ppm
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:01:58.450584 2437 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:01:58.451503 2437 webserver.cc:492] Webserver started at http://127.1.157.194:37631/ using document root <none> and password file <none>
I20251214 03:01:58.451982 2437 fs_manager.cc:362] Metadata directory not provided
I20251214 03:01:58.452026 2437 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:01:58.452190 2437 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:01:58.453795 2437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "59413ad5dd3147c8bcf14193ff9df5f9"
format_stamp: "Formatted at 2025-12-14 03:01:58 on dist-test-slave-k7z7"
server_key: "08d322da68b68fd044f5e9bcf504f341"
server_key_iv: "5145b6b3d48a4fb2353d01efd3ffa20d"
server_key_version: "encryptionkey@0"
I20251214 03:01:58.454221 2437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "59413ad5dd3147c8bcf14193ff9df5f9"
format_stamp: "Formatted at 2025-12-14 03:01:58 on dist-test-slave-k7z7"
server_key: "08d322da68b68fd044f5e9bcf504f341"
server_key_iv: "5145b6b3d48a4fb2353d01efd3ffa20d"
server_key_version: "encryptionkey@0"
I20251214 03:01:58.457279 2437 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:01:58.459528 2452 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:58.460435 2437 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:01:58.460520 2437 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "59413ad5dd3147c8bcf14193ff9df5f9"
format_stamp: "Formatted at 2025-12-14 03:01:58 on dist-test-slave-k7z7"
server_key: "08d322da68b68fd044f5e9bcf504f341"
server_key_iv: "5145b6b3d48a4fb2353d01efd3ffa20d"
server_key_version: "encryptionkey@0"
I20251214 03:01:58.460592 2437 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:01:58.507182 2437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:01:58.509917 2437 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:01:58.510078 2437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:01:58.510584 2437 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:01:58.511442 2437 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:01:58.511489 2437 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:58.511528 2437 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:01:58.511547 2437 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:58.520076 2437 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:42771
I20251214 03:01:58.520104 2565 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:42771 every 8 connection(s)
I20251214 03:01:58.520954 2437 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:01:58.524155 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 2437
I20251214 03:01:58.524271 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:01:58.524494 1655 external_mini_cluster.cc:1468] Setting key 22f908f0429ca5fa6edfc396df2ed96b
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
I20251214 03:01:58.533243 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.522590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:43991 (local address 127.1.157.254:46501)
1214 03:01:58.522735 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.522741 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.523362 (+ 621us) server_negotiation.cc:408] Connection header received
1214 03:01:58.524119 (+ 757us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.524123 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.524181 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.524280 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:58.526112 (+ 1832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.526762 (+ 650us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.527598 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.527777 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.529906 (+ 2129us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:58.529925 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:58.529927 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:58.529953 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:58.531357 (+ 1404us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.531827 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.531830 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.531831 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.531879 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.532133 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.532137 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.532139 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.532308 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:58.532399 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.532900 (+ 501us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.533029 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251214 03:01:58.533865 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.522822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:46501 (local address 127.1.157.194:43991)
1214 03:01:58.523228 (+ 406us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:58.523262 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:58.523940 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:58.524427 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:58.524435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:58.524866 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:01:58.525978 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.525992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.526901 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.526905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:58.527480 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.527490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.527708 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.528324 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:01:58.528347 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:01:58.529753 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:01:58.531475 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:58.531481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:58.531494 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:58.531733 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:58.531958 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:58.531962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:58.531963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:58.532053 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:58.532412 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:01:58.532420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:01:58.532661 (+ 241us) client_negotiation.cc:769] Sending connection context
1214 03:01:58.532851 (+ 190us) client_negotiation.cc:240] Negotiation successful
1214 03:01:58.533127 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
I20251214 03:01:58.534972 2566 heartbeater.cc:344] Connected to a master server at 127.1.157.254:46501
I20251214 03:01:58.535221 2566 heartbeater.cc:461] Registering TS with master...
I20251214 03:01:58.535741 2566 heartbeater.cc:507] Master 127.1.157.254:46501 requested a full tablet report, sending...
I20251214 03:01:58.536811 2231 ts_manager.cc:194] Registered new tserver with Master: 59413ad5dd3147c8bcf14193ff9df5f9 (127.1.157.194:42771)
I20251214 03:01:58.537475 2231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:43991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:01:58.569880 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:46501
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:01:58.671787 2573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:01:58.671972 2573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:01:58.672035 2573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:01:58.675163 2573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:01:58.675215 2573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:01:58.675304 2573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:01:58.679261 2573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:46501
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:01:58.680234 2573 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:01:58.680999 2573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:01:58.687546 2581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:58.687537 2579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:01:58.687501 2578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:01:58.687811 2573 server_base.cc:1047] running on GCE node
I20251214 03:01:58.688163 2573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:01:58.688705 2573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:01:58.689877 2573 hybrid_clock.cc:648] HybridClock initialized: now 1765681318689855 us; error 35 us; skew 500 ppm
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:01:58.692565 2573 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:01:58.693841 2573 webserver.cc:492] Webserver started at http://127.1.157.195:43405/ using document root <none> and password file <none>
I20251214 03:01:58.694535 2573 fs_manager.cc:362] Metadata directory not provided
I20251214 03:01:58.694591 2573 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:01:58.694762 2573 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:01:58.696431 2573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "c6467664ab4445ed982668ed50b24da7"
format_stamp: "Formatted at 2025-12-14 03:01:58 on dist-test-slave-k7z7"
server_key: "46f9008b4119a8a5a8d22176e65bc068"
server_key_iv: "ab2e0ea2dbe11360be4925baed9b0288"
server_key_version: "encryptionkey@0"
I20251214 03:01:58.696862 2573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "c6467664ab4445ed982668ed50b24da7"
format_stamp: "Formatted at 2025-12-14 03:01:58 on dist-test-slave-k7z7"
server_key: "46f9008b4119a8a5a8d22176e65bc068"
server_key_iv: "ab2e0ea2dbe11360be4925baed9b0288"
server_key_version: "encryptionkey@0"
I20251214 03:01:58.700219 2573 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251214 03:01:58.702414 2588 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:58.703447 2573 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:01:58.703547 2573 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "c6467664ab4445ed982668ed50b24da7"
format_stamp: "Formatted at 2025-12-14 03:01:58 on dist-test-slave-k7z7"
server_key: "46f9008b4119a8a5a8d22176e65bc068"
server_key_iv: "ab2e0ea2dbe11360be4925baed9b0288"
server_key_version: "encryptionkey@0"
I20251214 03:01:58.703644 2573 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:01:58.718776 2573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:01:58.721500 2573 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:01:58.721657 2573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:01:58.722164 2573 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:01:58.722967 2573 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:01:58.723013 2573 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:58.723066 2573 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:01:58.723094 2573 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:01:58.731630 2573 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:46455
I20251214 03:01:58.731662 2701 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:46455 every 8 connection(s)
I20251214 03:01:58.732515 2573 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:01:58.734764 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 2573
I20251214 03:01:58.734867 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:01:58.735090 1655 external_mini_cluster.cc:1468] Setting key 6cd32aa16b33828f82f80b5ccc71ea42
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:01:58.743752 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.734099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:45189 (local address 127.1.157.254:46501)
1214 03:01:58.734285 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.734290 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.734869 (+ 579us) server_negotiation.cc:408] Connection header received
1214 03:01:58.735797 (+ 928us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.735799 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.735850 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.735926 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:58.737666 (+ 1740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.738151 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.738780 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.738907 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.741010 (+ 2103us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:58.741024 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:58.741026 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:58.741049 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:58.742222 (+ 1173us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.742685 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.742687 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.742689 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.742721 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.742918 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.742919 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.742921 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.743042 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:58.743105 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.743551 (+ 446us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.743638 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251214 03:01:58.744329 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.734352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:46501 (local address 127.1.157.195:45189)
1214 03:01:58.734750 (+ 398us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:58.734784 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:58.735609 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:58.736105 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:58.736115 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:58.736551 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:01:58.737539 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.737554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.738275 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.738279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:58.738682 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.738690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.738864 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.739560 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:01:58.739584 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:01:58.740890 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:01:58.742369 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:58.742374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:58.742385 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:58.742605 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:58.742780 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:01:58.742783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:01:58.742784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:01:58.742858 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:01:58.743142 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:01:58.743148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:01:58.743355 (+ 207us) client_negotiation.cc:769] Sending connection context
1214 03:01:58.743515 (+ 160us) client_negotiation.cc:240] Negotiation successful
1214 03:01:58.743701 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
I20251214 03:01:58.745350 2702 heartbeater.cc:344] Connected to a master server at 127.1.157.254:46501
I20251214 03:01:58.745560 2702 heartbeater.cc:461] Registering TS with master...
I20251214 03:01:58.746012 2702 heartbeater.cc:507] Master 127.1.157.254:46501 requested a full tablet report, sending...
I20251214 03:01:58.746937 2231 ts_manager.cc:194] Registered new tserver with Master: c6467664ab4445ed982668ed50b24da7 (127.1.157.195:46455)
I20251214 03:01:58.747511 2231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:45189
I20251214 03:01:58.748459 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251214 03:01:58.756552 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.751073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.1.157.254:46501)
1214 03:01:58.751189 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.751192 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.751322 (+ 130us) server_negotiation.cc:408] Connection header received
1214 03:01:58.751411 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.751414 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.751448 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.751512 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:58.752190 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.752660 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.753253 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.753370 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.754299 (+ 929us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:58.754311 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:58.754314 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:58.754327 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:58.755447 (+ 1120us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.755767 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.755770 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.755771 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.755807 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.755952 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.755954 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.755955 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.756102 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:58.756193 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.756350 (+ 157us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.756438 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251214 03:01:58.760182 2231 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:01:58.762564 2231 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:01:58.775041 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.771106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:42771 (local address 127.0.0.1:33354)
1214 03:01:58.771340 (+ 234us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:58.771353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:58.771443 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:58.772366 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:58.772371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:58.772392 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:01:58.772713 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.772722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.773867 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.773870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:58.774582 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.774591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.774689 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.774740 (+ 51us) client_negotiation.cc:769] Sending connection context
1214 03:01:58.774819 (+ 79us) client_negotiation.cc:240] Negotiation successful
1214 03:01:58.774902 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20251214 03:01:58.775527 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.770595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:35923 (local address 127.0.0.1:45422)
1214 03:01:58.771225 (+ 630us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:58.771263 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:58.771387 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:58.772769 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:58.772773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:58.772792 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:01:58.773054 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.773063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.774353 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.774357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:58.775188 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.775198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.775289 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.775304 (+ 15us) client_negotiation.cc:769] Sending connection context
1214 03:01:58.775342 (+ 38us) client_negotiation.cc:240] Negotiation successful
1214 03:01:58.775401 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":68,"threads_started":1}
I20251214 03:01:58.775776 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.771930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:46455 (local address 127.0.0.1:38236)
1214 03:01:58.772136 (+ 206us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:58.772148 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:58.772219 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:58.773132 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:58.773136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:58.773154 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:01:58.773432 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.773439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.774624 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.774626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:58.775424 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.775433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.775535 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.775553 (+ 18us) client_negotiation.cc:769] Sending connection context
1214 03:01:58.775597 (+ 44us) client_negotiation.cc:240] Negotiation successful
1214 03:01:58.775650 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20251214 03:01:58.776340 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.772085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.1.157.195:46455)
1214 03:01:58.772758 (+ 673us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.772764 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.772783 (+ 19us) server_negotiation.cc:408] Connection header received
1214 03:01:58.772856 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.772861 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.773006 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.773136 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:01:58.773542 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.774526 (+ 984us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.775553 (+ 1027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.776034 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.776119 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.776166 (+ 47us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.776232 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":143,"threads_started":1}
I20251214 03:01:58.776703 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.770913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.1.157.193:35923)
1214 03:01:58.772372 (+ 1459us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.772378 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.772398 (+ 20us) server_negotiation.cc:408] Connection header received
1214 03:01:58.772473 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.772478 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.772637 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.772765 (+ 128us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:01:58.773204 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.774240 (+ 1036us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.775781 (+ 1541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.776313 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.776410 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.776474 (+ 64us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.776571 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1362,"thread_start_us":96,"threads_started":1}
I20251214 03:01:58.776922 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.771591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.1.157.194:42771)
1214 03:01:58.771904 (+ 313us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.771908 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.771927 (+ 19us) server_negotiation.cc:408] Connection header received
1214 03:01:58.771999 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.772005 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.772182 (+ 177us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.772365 (+ 183us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:01:58.772839 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.773773 (+ 934us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.775781 (+ 2008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.776573 (+ 792us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.776659 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.776719 (+ 60us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.776802 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":95,"threads_started":1}
I20251214 03:01:58.778884 2636 tablet_service.cc:1505] Processing CreateTablet for tablet 7acf13340d254645927ab23e4cef43e2 (DEFAULT_TABLE table=test-table [id=ed02a886d1e3465a83c4d912e0f92187]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:01:58.779222 2364 tablet_service.cc:1505] Processing CreateTablet for tablet 7acf13340d254645927ab23e4cef43e2 (DEFAULT_TABLE table=test-table [id=ed02a886d1e3465a83c4d912e0f92187]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:01:58.779220 2500 tablet_service.cc:1505] Processing CreateTablet for tablet 7acf13340d254645927ab23e4cef43e2 (DEFAULT_TABLE table=test-table [id=ed02a886d1e3465a83c4d912e0f92187]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:01:58.779739 2636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7acf13340d254645927ab23e4cef43e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:01:58.780102 2500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7acf13340d254645927ab23e4cef43e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:01:58.780102 2364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7acf13340d254645927ab23e4cef43e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:01:58.785030 2719 tablet_bootstrap.cc:492] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7: Bootstrap starting.
I20251214 03:01:58.786648 2721 tablet_bootstrap.cc:492] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9: Bootstrap starting.
I20251214 03:01:58.787317 2719 tablet_bootstrap.cc:654] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7: Neither blocks nor log segments found. Creating new log.
I20251214 03:01:58.788863 2719 log.cc:826] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7: Log is configured to *not* fsync() on all Append() calls
I20251214 03:01:58.788924 2721 tablet_bootstrap.cc:654] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9: Neither blocks nor log segments found. Creating new log.
I20251214 03:01:58.789592 2720 tablet_bootstrap.cc:492] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b: Bootstrap starting.
I20251214 03:01:58.789952 2721 log.cc:826] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9: Log is configured to *not* fsync() on all Append() calls
I20251214 03:01:58.792044 2720 tablet_bootstrap.cc:654] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b: Neither blocks nor log segments found. Creating new log.
I20251214 03:01:58.792140 2721 tablet_bootstrap.cc:492] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9: No bootstrap required, opened a new log
I20251214 03:01:58.792353 2721 ts_tablet_manager.cc:1403] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20251214 03:01:58.793460 2719 tablet_bootstrap.cc:492] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7: No bootstrap required, opened a new log
I20251214 03:01:58.793661 2719 ts_tablet_manager.cc:1403] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20251214 03:01:58.793996 2720 log.cc:826] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b: Log is configured to *not* fsync() on all Append() calls
I20251214 03:01:58.796501 2719 raft_consensus.cc:359] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff21570316434a33b8d2d0ccd68b374b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 35923 } } peers { permanent_uuid: "c6467664ab4445ed982668ed50b24da7" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46455 } } peers { permanent_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 42771 } }
I20251214 03:01:58.796693 2719 raft_consensus.cc:385] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:01:58.796748 2719 raft_consensus.cc:740] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6467664ab4445ed982668ed50b24da7, State: Initialized, Role: FOLLOWER
I20251214 03:01:58.796710 2721 raft_consensus.cc:359] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff21570316434a33b8d2d0ccd68b374b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 35923 } } peers { permanent_uuid: "c6467664ab4445ed982668ed50b24da7" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46455 } } peers { permanent_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 42771 } }
I20251214 03:01:58.796912 2721 raft_consensus.cc:385] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:01:58.796973 2721 raft_consensus.cc:740] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59413ad5dd3147c8bcf14193ff9df5f9, State: Initialized, Role: FOLLOWER
I20251214 03:01:58.797273 2719 consensus_queue.cc:260] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff21570316434a33b8d2d0ccd68b374b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 35923 } } peers { permanent_uuid: "c6467664ab4445ed982668ed50b24da7" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46455 } } peers { permanent_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 42771 } }
I20251214 03:01:58.797392 2721 consensus_queue.cc:260] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff21570316434a33b8d2d0ccd68b374b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 35923 } } peers { permanent_uuid: "c6467664ab4445ed982668ed50b24da7" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46455 } } peers { permanent_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 42771 } }
I20251214 03:01:58.798130 2566 heartbeater.cc:499] Master 127.1.157.254:46501 was elected leader, sending a full tablet report...
I20251214 03:01:58.798194 2719 ts_tablet_manager.cc:1434] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251214 03:01:58.798537 2702 heartbeater.cc:499] Master 127.1.157.254:46501 was elected leader, sending a full tablet report...
I20251214 03:01:58.799698 2721 ts_tablet_manager.cc:1434] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251214 03:01:58.803056 2720 tablet_bootstrap.cc:492] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b: No bootstrap required, opened a new log
I20251214 03:01:58.803376 2720 ts_tablet_manager.cc:1403] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20251214 03:01:58.806416 2720 raft_consensus.cc:359] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff21570316434a33b8d2d0ccd68b374b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 35923 } } peers { permanent_uuid: "c6467664ab4445ed982668ed50b24da7" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46455 } } peers { permanent_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 42771 } }
I20251214 03:01:58.806658 2720 raft_consensus.cc:385] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:01:58.806730 2720 raft_consensus.cc:740] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff21570316434a33b8d2d0ccd68b374b, State: Initialized, Role: FOLLOWER
I20251214 03:01:58.807206 2720 consensus_queue.cc:260] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff21570316434a33b8d2d0ccd68b374b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 35923 } } peers { permanent_uuid: "c6467664ab4445ed982668ed50b24da7" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46455 } } peers { permanent_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 42771 } }
I20251214 03:01:58.807497 2726 raft_consensus.cc:493] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:01:58.807644 2726 raft_consensus.cc:515] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff21570316434a33b8d2d0ccd68b374b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 35923 } } peers { permanent_uuid: "c6467664ab4445ed982668ed50b24da7" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46455 } } peers { permanent_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 42771 } }
I20251214 03:01:58.808164 2430 heartbeater.cc:499] Master 127.1.157.254:46501 was elected leader, sending a full tablet report...
I20251214 03:01:58.808207 2720 ts_tablet_manager.cc:1434] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251214 03:01:58.808892 2726 leader_election.cc:290] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff21570316434a33b8d2d0ccd68b374b (127.1.157.193:35923), c6467664ab4445ed982668ed50b24da7 (127.1.157.195:46455)
I20251214 03:01:58.812222 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.809097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:35923 (local address 127.1.157.194:54087)
1214 03:01:58.809291 (+ 194us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:58.809305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:58.809393 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:58.809754 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:58.809758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:58.809781 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:01:58.810043 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.810052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.810961 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.810965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:58.811849 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.811860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.811965 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.811989 (+ 24us) client_negotiation.cc:769] Sending connection context
1214 03:01:58.812036 (+ 47us) client_negotiation.cc:240] Negotiation successful
1214 03:01:58.812094 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"spinlock_wait_cycles":20736}
I20251214 03:01:58.813203 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.809406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:46455 (local address 127.1.157.194:44279)
1214 03:01:58.809675 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1214 03:01:58.809688 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:01:58.809833 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:01:58.810183 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:01:58.810186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:01:58.810203 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:01:58.810495 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.810504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.811444 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.811447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:01:58.812329 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:01:58.812340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.812917 (+ 577us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.812930 (+ 13us) client_negotiation.cc:769] Sending connection context
1214 03:01:58.812996 (+ 66us) client_negotiation.cc:240] Negotiation successful
1214 03:01:58.813050 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
I20251214 03:01:58.813211 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.809300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:44279 (local address 127.1.157.195:46455)
1214 03:01:58.809400 (+ 100us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.809404 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.809751 (+ 347us) server_negotiation.cc:408] Connection header received
1214 03:01:58.809962 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.809966 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.810012 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.810058 (+ 46us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:01:58.810619 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.811312 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.812457 (+ 1145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.812831 (+ 374us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.812866 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.813031 (+ 165us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.813092 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251214 03:01:58.813217 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.809144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:54087 (local address 127.1.157.193:35923)
1214 03:01:58.809309 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.809313 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.809483 (+ 170us) server_negotiation.cc:408] Connection header received
1214 03:01:58.809528 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.809531 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.809577 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.809642 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:01:58.810158 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.810836 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.812508 (+ 1672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.812956 (+ 448us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.812990 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.813037 (+ 47us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.813088 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20251214 03:01:58.813961 2384 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7acf13340d254645927ab23e4cef43e2" candidate_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff21570316434a33b8d2d0ccd68b374b" is_pre_election: true
I20251214 03:01:58.814064 2656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7acf13340d254645927ab23e4cef43e2" candidate_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6467664ab4445ed982668ed50b24da7" is_pre_election: true
I20251214 03:01:58.814183 2384 raft_consensus.cc:2468] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59413ad5dd3147c8bcf14193ff9df5f9 in term 0.
I20251214 03:01:58.814231 2656 raft_consensus.cc:2468] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59413ad5dd3147c8bcf14193ff9df5f9 in term 0.
I20251214 03:01:58.814597 2456 leader_election.cc:304] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59413ad5dd3147c8bcf14193ff9df5f9, c6467664ab4445ed982668ed50b24da7; no voters:
I20251214 03:01:58.814844 2726 raft_consensus.cc:2804] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:01:58.814919 2726 raft_consensus.cc:493] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:01:58.814966 2726 raft_consensus.cc:3060] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:01:58.816099 2726 raft_consensus.cc:515] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff21570316434a33b8d2d0ccd68b374b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 35923 } } peers { permanent_uuid: "c6467664ab4445ed982668ed50b24da7" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46455 } } peers { permanent_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 42771 } }
I20251214 03:01:58.816470 2726 leader_election.cc:290] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [CANDIDATE]: Term 1 election: Requested vote from peers ff21570316434a33b8d2d0ccd68b374b (127.1.157.193:35923), c6467664ab4445ed982668ed50b24da7 (127.1.157.195:46455)
I20251214 03:01:58.816744 2384 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7acf13340d254645927ab23e4cef43e2" candidate_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff21570316434a33b8d2d0ccd68b374b"
I20251214 03:01:58.816849 2384 raft_consensus.cc:3060] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:01:58.817761 2384 raft_consensus.cc:2468] T 7acf13340d254645927ab23e4cef43e2 P ff21570316434a33b8d2d0ccd68b374b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59413ad5dd3147c8bcf14193ff9df5f9 in term 1.
I20251214 03:01:58.818050 2454 leader_election.cc:304] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59413ad5dd3147c8bcf14193ff9df5f9, ff21570316434a33b8d2d0ccd68b374b; no voters:
I20251214 03:01:58.818161 2656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7acf13340d254645927ab23e4cef43e2" candidate_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6467664ab4445ed982668ed50b24da7"
I20251214 03:01:58.818287 2656 raft_consensus.cc:3060] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:01:58.818396 2726 raft_consensus.cc:2804] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:01:58.818591 2726 raft_consensus.cc:697] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 59413ad5dd3147c8bcf14193ff9df5f9, State: Running, Role: LEADER
I20251214 03:01:58.818885 2726 consensus_queue.cc:237] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff21570316434a33b8d2d0ccd68b374b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 35923 } } peers { permanent_uuid: "c6467664ab4445ed982668ed50b24da7" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46455 } } peers { permanent_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 42771 } }
I20251214 03:01:58.819315 2656 raft_consensus.cc:2468] T 7acf13340d254645927ab23e4cef43e2 P c6467664ab4445ed982668ed50b24da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59413ad5dd3147c8bcf14193ff9df5f9 in term 1.
I20251214 03:01:58.821883 2237 catalog_manager.cc:5654] T 7acf13340d254645927ab23e4cef43e2 P 59413ad5dd3147c8bcf14193ff9df5f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59413ad5dd3147c8bcf14193ff9df5f9 (127.1.157.194). New cstate: current_term: 1 leader_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff21570316434a33b8d2d0ccd68b374b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 35923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6467664ab4445ed982668ed50b24da7" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59413ad5dd3147c8bcf14193ff9df5f9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 42771 } health_report { overall_health: HEALTHY } } }
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.193@KRBTEST.COM
I20251214 03:01:58.837983 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.830228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.1.157.193:35923)
1214 03:01:58.830357 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.830362 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.830521 (+ 159us) server_negotiation.cc:408] Connection header received
1214 03:01:58.830630 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.830634 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.830676 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.830737 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:58.831504 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.831958 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.832557 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.832730 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.835017 (+ 2287us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:58.835037 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:58.835048 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:58.835072 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:58.836599 (+ 1527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.836952 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.836957 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.836958 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.837011 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.837223 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.837226 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.837229 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.837394 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:58.837611 (+ 217us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.837734 (+ 123us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.837846 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.157.193@KRBTEST.COM
I20251214 03:01:58.861095 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.853587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.1.157.193:35923)
1214 03:01:58.853733 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.853738 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.853827 (+ 89us) server_negotiation.cc:408] Connection header received
1214 03:01:58.853964 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.853967 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.854013 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.854074 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:58.854842 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.855327 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.855946 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.856132 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.858277 (+ 2145us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:58.858298 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:58.858301 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:58.858331 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:58.859743 (+ 1412us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.860109 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.860112 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.860114 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.860159 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.860354 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.860357 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.860359 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.860502 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:58.860616 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.860834 (+ 218us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.860942 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.193@KRBTEST.COM
I20251214 03:01:58.883401 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:01:58.876026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.1.157.193:35923)
1214 03:01:58.876174 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1214 03:01:58.876178 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:01:58.876292 (+ 114us) server_negotiation.cc:408] Connection header received
1214 03:01:58.876414 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:01:58.876417 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:01:58.876464 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:01:58.876538 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:01:58.877299 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.877754 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:01:58.878437 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:01:58.878582 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:01:58.880720 (+ 2138us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:01:58.880740 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:01:58.880742 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:01:58.880769 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:01:58.882181 (+ 1412us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.882557 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.882560 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.882562 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.882598 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:01:58.882763 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:01:58.882765 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:01:58.882767 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:01:58.882900 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:01:58.882999 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1214 03:01:58.883149 (+ 150us) server_negotiation.cc:300] Negotiation successful
1214 03:01:58.883248 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20251214 03:01:58.884687 2344 tablet_service.cc:3012] Rejecting scan request for tablet 7acf13340d254645927ab23e4cef43e2: Uninitialized: clean time has not yet been initialized
I20251214 03:01:58.886293 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 2301
I20251214 03:01:58.891537 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 2437
I20251214 03:01:58.896102 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 2573
I20251214 03:01:58.900403 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 2205
2025-12-14T03:01:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3038 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2743](info): set up 2 sockets
Dec 14 03:01:58 dist-test-slave-k7z7 krb5kdc[2743](info): commencing operation
krb5kdc: starting...
W20251214 03:02:00.732975 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.811s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:02:00 dist-test-slave-k7z7 krb5kdc[2743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:02:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:00.869966 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:33877
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:33877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:00.974483 2759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:00.974707 2759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:00.974781 2759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:00.978135 2759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:00.978202 2759 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251214 03:02:00.978226 2759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:00.978255 2759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:00.978286 2759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:00.978317 2759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:00.982528 2759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:33877
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:33877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:00.983515 2759 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:00.984284 2759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:00.989665 2765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:00.989748 2759 server_base.cc:1047] running on GCE node
W20251214 03:02:00.989712 2767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:00.989670 2764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:00.990365 2759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:00.991201 2759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:00.992347 2759 hybrid_clock.cc:648] HybridClock initialized: now 1765681320992338 us; error 25 us; skew 500 ppm
Dec 14 03:02:00 dist-test-slave-k7z7 krb5kdc[2743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681320, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:00.994683 2759 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:02:00.995684 2759 webserver.cc:492] Webserver started at http://127.1.157.254:35259/ using document root <none> and password file <none>
I20251214 03:02:00.996177 2759 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:00.996225 2759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:00.996398 2759 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:00.998000 2759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "169c7196f42b43dd8aa03cf57da7f9e0"
format_stamp: "Formatted at 2025-12-14 03:02:00 on dist-test-slave-k7z7"
server_key: "43356b0453768b0af44f3753d0632259"
server_key_iv: "290bd3665bcf4cc5f976ae51fa94b1ec"
server_key_version: "encryptionkey@0"
I20251214 03:02:00.998580 2759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "169c7196f42b43dd8aa03cf57da7f9e0"
format_stamp: "Formatted at 2025-12-14 03:02:00 on dist-test-slave-k7z7"
server_key: "43356b0453768b0af44f3753d0632259"
server_key_iv: "290bd3665bcf4cc5f976ae51fa94b1ec"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.001652 2759 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251214 03:02:01.003618 2774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:01.004537 2759 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:01.004634 2759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "169c7196f42b43dd8aa03cf57da7f9e0"
format_stamp: "Formatted at 2025-12-14 03:02:00 on dist-test-slave-k7z7"
server_key: "43356b0453768b0af44f3753d0632259"
server_key_iv: "290bd3665bcf4cc5f976ae51fa94b1ec"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.004709 2759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:01.016984 2759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:01.022552 2759 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:01.022698 2759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:01.029963 2836 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:33877 every 8 connection(s)
I20251214 03:02:01.029963 2759 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:33877
I20251214 03:02:01.030893 2759 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:01.033516 2837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:01.035665 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 2759
I20251214 03:02:01.035740 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:01.035923 1655 external_mini_cluster.cc:1468] Setting key 691f412e795ca120de651d79fa490873
I20251214 03:02:01.038910 2837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0: Bootstrap starting.
I20251214 03:02:01.041311 2837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:01.042093 2837 log.cc:826] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0: Log is configured to *not* fsync() on all Append() calls
Dec 14 03:02:01 dist-test-slave-k7z7 krb5kdc[2743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:01.043747 2837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0: No bootstrap required, opened a new log
I20251214 03:02:01.046171 2837 raft_consensus.cc:359] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169c7196f42b43dd8aa03cf57da7f9e0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33877 } }
I20251214 03:02:01.046336 2837 raft_consensus.cc:385] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:01.046391 2837 raft_consensus.cc:740] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169c7196f42b43dd8aa03cf57da7f9e0, State: Initialized, Role: FOLLOWER
I20251214 03:02:01.046784 2837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169c7196f42b43dd8aa03cf57da7f9e0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33877 } }
I20251214 03:02:01.046909 2837 raft_consensus.cc:399] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:01.046962 2837 raft_consensus.cc:493] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:01.047045 2837 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:01.047766 2837 raft_consensus.cc:515] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169c7196f42b43dd8aa03cf57da7f9e0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33877 } }
I20251214 03:02:01.048082 2837 leader_election.cc:304] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 169c7196f42b43dd8aa03cf57da7f9e0; no voters:
I20251214 03:02:01.048350 2837 leader_election.cc:290] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:01.048451 2842 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:01.048511 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.037012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.1.157.254:33877)
1214 03:02:01.037418 (+ 406us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.037429 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.037469 (+ 40us) server_negotiation.cc:408] Connection header received
1214 03:02:01.038087 (+ 618us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.038111 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.038455 (+ 344us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.038783 (+ 328us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:01.039573 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.040698 (+ 1125us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.041404 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.041679 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.043722 (+ 2043us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:01.043739 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:01.043749 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:01.043780 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:01.045941 (+ 2161us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.046503 (+ 562us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.046508 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.046512 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.046579 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.046792 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.046796 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.046798 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.047167 (+ 369us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:01.047366 (+ 199us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.047621 (+ 255us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.047855 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
I20251214 03:02:01.048636 2842 raft_consensus.cc:697] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [term 1 LEADER]: Becoming Leader. State: Replica: 169c7196f42b43dd8aa03cf57da7f9e0, State: Running, Role: LEADER
I20251214 03:02:01.048900 2842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169c7196f42b43dd8aa03cf57da7f9e0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33877 } }
I20251214 03:02:01.049368 2837 sys_catalog.cc:565] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:01.050006 2842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169c7196f42b43dd8aa03cf57da7f9e0. Latest consensus state: current_term: 1 leader_uuid: "169c7196f42b43dd8aa03cf57da7f9e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169c7196f42b43dd8aa03cf57da7f9e0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33877 } } }
I20251214 03:02:01.050110 2842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:01.050403 2843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "169c7196f42b43dd8aa03cf57da7f9e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169c7196f42b43dd8aa03cf57da7f9e0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33877 } } }
I20251214 03:02:01.050446 2849 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:01.050498 2843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:01.053226 2849 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:01.058056 2849 catalog_manager.cc:1357] Generated new cluster ID: 8be0c3d4bca04544b4e27680ce793529
I20251214 03:02:01.058120 2849 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:01.089815 2849 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:01.091126 2849 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:01.102425 2849 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0: Generated new TSK 0
I20251214 03:02:01.103240 2849 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:01.149899 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:33877
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:01.253062 2865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:01.253288 2865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:01.253365 2865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:01.256494 2865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:01.256551 2865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:01.256619 2865 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251214 03:02:01.256678 2865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:01.260661 2865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.157.254:33877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:01.261636 2865 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:01.262406 2865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:01.268447 2871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:01.268337 2870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:01.268447 2873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:01.268828 2865 server_base.cc:1047] running on GCE node
I20251214 03:02:01.269202 2865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:01.269695 2865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:01.270838 2865 hybrid_clock.cc:648] HybridClock initialized: now 1765681321270826 us; error 26 us; skew 500 ppm
Dec 14 03:02:01 dist-test-slave-k7z7 krb5kdc[2743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:01.273195 2865 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:02:01.274093 2865 webserver.cc:492] Webserver started at http://127.1.157.193:32779/ using document root <none> and password file <none>
I20251214 03:02:01.274590 2865 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:01.274636 2865 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:01.274787 2865 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:01.276364 2865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "cf9b7c7511984e87afd03e4bfbfc6c95"
format_stamp: "Formatted at 2025-12-14 03:02:01 on dist-test-slave-k7z7"
server_key: "da7ec0388c7bd910ec19086d8af49243"
server_key_iv: "67ea13a3d284640aa4a08c40f32705bb"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.276773 2865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "cf9b7c7511984e87afd03e4bfbfc6c95"
format_stamp: "Formatted at 2025-12-14 03:02:01 on dist-test-slave-k7z7"
server_key: "da7ec0388c7bd910ec19086d8af49243"
server_key_iv: "67ea13a3d284640aa4a08c40f32705bb"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.279783 2865 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251214 03:02:01.281858 2880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:01.282802 2865 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:01.282894 2865 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "cf9b7c7511984e87afd03e4bfbfc6c95"
format_stamp: "Formatted at 2025-12-14 03:02:01 on dist-test-slave-k7z7"
server_key: "da7ec0388c7bd910ec19086d8af49243"
server_key_iv: "67ea13a3d284640aa4a08c40f32705bb"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.282982 2865 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:01.316133 2865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:01.318966 2865 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:01.319144 2865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:01.320624 2865 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:01.320679 2865 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:01.320734 2865 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:01.320761 2865 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 14 03:02:01 dist-test-slave-k7z7 krb5kdc[2743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:01.331029 2865 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:34795
I20251214 03:02:01.331050 2995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:34795 every 8 connection(s)
I20251214 03:02:01.332113 2865 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:02:01.332635 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.321321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:49357 (local address 127.1.157.254:33877)
1214 03:02:01.321439 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.321443 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.322297 (+ 854us) server_negotiation.cc:408] Connection header received
1214 03:02:01.323131 (+ 834us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.323135 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.323197 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.323268 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:01.324851 (+ 1583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.325567 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.326258 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.326438 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.328888 (+ 2450us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:01.328913 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:01.328916 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:01.328947 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:01.330598 (+ 1651us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.331244 (+ 646us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.331248 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.331250 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.331302 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.331584 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.331587 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.331589 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.331779 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:01.331884 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.332316 (+ 432us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.332467 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251214 03:02:01.333235 2889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.321660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33877 (local address 127.1.157.193:49357)
1214 03:02:01.322161 (+ 501us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:01.322196 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:01.322937 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:01.323460 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:01.323470 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:01.323980 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:01.324667 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.324684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.325688 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.325692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:01.326145 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.326153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.326375 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.327011 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:01.327048 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:01.328744 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:01.330738 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.330744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.330755 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.331083 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.331413 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.331416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.331417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.331503 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.331870 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:01.331875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:01.332088 (+ 213us) client_negotiation.cc:769] Sending connection context
1214 03:02:01.332279 (+ 191us) client_negotiation.cc:240] Negotiation successful
1214 03:02:01.332515 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":169,"threads_started":1}
I20251214 03:02:01.334278 2996 heartbeater.cc:344] Connected to a master server at 127.1.157.254:33877
I20251214 03:02:01.334422 2996 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:01.334894 2996 heartbeater.cc:507] Master 127.1.157.254:33877 requested a full tablet report, sending...
I20251214 03:02:01.335568 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 2865
I20251214 03:02:01.335660 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:02:01.335877 1655 external_mini_cluster.cc:1468] Setting key f054ea12a651f33ac6332247a0deb869
I20251214 03:02:01.336390 2790 ts_manager.cc:194] Registered new tserver with Master: cf9b7c7511984e87afd03e4bfbfc6c95 (127.1.157.193:34795)
I20251214 03:02:01.337750 2790 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:49357
I20251214 03:02:01.342361 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.336239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:58045 (local address 127.1.157.254:33877)
1214 03:02:01.336452 (+ 213us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.336455 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.336534 (+ 79us) server_negotiation.cc:408] Connection header received
1214 03:02:01.336647 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.336650 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.336690 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.336766 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:01.337363 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.338013 (+ 650us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.338747 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.338931 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.339689 (+ 758us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:01.339704 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:01.339708 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:01.339730 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:01.341005 (+ 1275us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.341419 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.341422 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.341424 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.341465 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.341664 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.341667 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.341669 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.341803 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:01.341885 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.342120 (+ 235us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.342233 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20251214 03:02:01.342372 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.336165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33877 (local address 127.1.157.193:58045)
1214 03:02:01.336465 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:01.336480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:01.336571 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:01.336776 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:01.336779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:01.336945 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:01.337230 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.337239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.338139 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.338142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:01.338643 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.338652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.338753 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.339294 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:01.339306 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:01.339578 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:01.341110 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.341113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.341116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.341328 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.341537 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.341541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.341543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.341594 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.341884 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:01.341887 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:01.341974 (+ 87us) client_negotiation.cc:769] Sending connection context
1214 03:02:01.342060 (+ 86us) client_negotiation.cc:240] Negotiation successful
1214 03:02:01.342187 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:01.380069 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:33877
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:01.487178 3007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:01.487423 3007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:01.487497 3007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:01.490921 3007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:01.491019 3007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:01.491076 3007 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251214 03:02:01.491125 3007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:01.495446 3007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.157.254:33877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:01.496519 3007 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:01.497349 3007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:01.503859 3015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:01.503970 3007 server_base.cc:1047] running on GCE node
W20251214 03:02:01.503855 3012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:01.503867 3013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:01.504467 3007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:01.504962 3007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:01.506105 3007 hybrid_clock.cc:648] HybridClock initialized: now 1765681321506081 us; error 43 us; skew 500 ppm
Dec 14 03:02:01 dist-test-slave-k7z7 krb5kdc[2743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:01.508600 3007 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:02:01.509691 3007 webserver.cc:492] Webserver started at http://127.1.157.194:40657/ using document root <none> and password file <none>
I20251214 03:02:01.510218 3007 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:01.510277 3007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:01.510449 3007 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:01.512158 3007 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "175c4087c31945aa95ce73d2f507fe4f"
format_stamp: "Formatted at 2025-12-14 03:02:01 on dist-test-slave-k7z7"
server_key: "4cc675dc321f1a42764553d7a384ca97"
server_key_iv: "bd2a7e6851362987cd9592bcc21c457c"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.512635 3007 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "175c4087c31945aa95ce73d2f507fe4f"
format_stamp: "Formatted at 2025-12-14 03:02:01 on dist-test-slave-k7z7"
server_key: "4cc675dc321f1a42764553d7a384ca97"
server_key_iv: "bd2a7e6851362987cd9592bcc21c457c"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.516134 3007 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251214 03:02:01.518365 3022 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:01.519444 3007 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:01.519556 3007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "175c4087c31945aa95ce73d2f507fe4f"
format_stamp: "Formatted at 2025-12-14 03:02:01 on dist-test-slave-k7z7"
server_key: "4cc675dc321f1a42764553d7a384ca97"
server_key_iv: "bd2a7e6851362987cd9592bcc21c457c"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.519646 3007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:01.551409 3007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:01.554173 3007 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:01.554338 3007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:01.555790 3007 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:01.555841 3007 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:01.555894 3007 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:01.555920 3007 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 14 03:02:01 dist-test-slave-k7z7 krb5kdc[2743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:01.566169 3007 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:36031
I20251214 03:02:01.566181 3137 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:36031 every 8 connection(s)
I20251214 03:02:01.567039 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.556391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:56079 (local address 127.1.157.254:33877)
1214 03:02:01.556541 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.556546 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.557231 (+ 685us) server_negotiation.cc:408] Connection header received
1214 03:02:01.558096 (+ 865us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.558099 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.558157 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.558226 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:01.559547 (+ 1321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.560161 (+ 614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.560848 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.561018 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.563490 (+ 2472us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:01.563520 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:01.563524 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:01.563557 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:01.565242 (+ 1685us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.565664 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.565669 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.565671 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.565727 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.566019 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.566022 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.566023 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.566169 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:01.566263 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.566754 (+ 491us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.566883 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251214 03:02:01.567114 3007 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:02:01.567664 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.556667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33877 (local address 127.1.157.194:56079)
1214 03:02:01.557089 (+ 422us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:01.557131 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:01.557904 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:01.558403 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:01.558413 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:01.558853 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:01.559394 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.559407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.560272 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.560276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:01.560740 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.560748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.560951 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.561597 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:01.561636 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:01.563334 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:01.565342 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.565346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.565355 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.565566 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.565814 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.565818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.565820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.565913 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.566266 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:01.566273 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:01.566530 (+ 257us) client_negotiation.cc:769] Sending connection context
1214 03:02:01.566705 (+ 175us) client_negotiation.cc:240] Negotiation successful
1214 03:02:01.566962 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20251214 03:02:01.569085 3138 heartbeater.cc:344] Connected to a master server at 127.1.157.254:33877
I20251214 03:02:01.569285 3138 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:01.569810 3138 heartbeater.cc:507] Master 127.1.157.254:33877 requested a full tablet report, sending...
I20251214 03:02:01.570742 2791 ts_manager.cc:194] Registered new tserver with Master: 175c4087c31945aa95ce73d2f507fe4f (127.1.157.194:36031)
I20251214 03:02:01.571434 2791 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:56079
I20251214 03:02:01.575282 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 3007
I20251214 03:02:01.575397 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:02:01.575615 1655 external_mini_cluster.cc:1468] Setting key 66ec5ff6183530685c6f79fd89aee0bd
I20251214 03:02:01.576702 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.570969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:59313 (local address 127.1.157.254:33877)
1214 03:02:01.571094 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.571098 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.571292 (+ 194us) server_negotiation.cc:408] Connection header received
1214 03:02:01.571392 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.571395 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.571454 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.571530 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:01.572112 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.572590 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.573160 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.573358 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.574060 (+ 702us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:01.574076 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:01.574079 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:01.574100 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:01.575234 (+ 1134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.575659 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.575662 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.575664 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.575715 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.575925 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.575929 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.575930 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.576100 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:01.576189 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.576476 (+ 287us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.576577 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251214 03:02:01.576716 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.570920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33877 (local address 127.1.157.194:59313)
1214 03:02:01.571222 (+ 302us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:01.571234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:01.571316 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:01.571590 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:01.571594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:01.571791 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:01.571998 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.572005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.572688 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.572691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:01.573063 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.573069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.573166 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.573684 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:01.573696 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:01.573950 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:01.575357 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.575361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.575362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.575563 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.575799 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.575801 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.575802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.575850 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.576200 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:01.576203 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:01.576298 (+ 95us) client_negotiation.cc:769] Sending connection context
1214 03:02:01.576398 (+ 100us) client_negotiation.cc:240] Negotiation successful
1214 03:02:01.576532 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:01.621614 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:33877
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:01.723865 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:01.724087 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:01.724154 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:01.727510 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:01.727588 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:01.727643 3149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251214 03:02:01.727692 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:01.731994 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.157.254:33877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:01.733120 3149 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:01.734025 3149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:01.740916 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:01.741004 3149 server_base.cc:1047] running on GCE node
W20251214 03:02:01.741024 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:01.740911 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:01.741503 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:01.742029 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:01.743165 3149 hybrid_clock.cc:648] HybridClock initialized: now 1765681321743159 us; error 23 us; skew 500 ppm
Dec 14 03:02:01 dist-test-slave-k7z7 krb5kdc[2743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:01.745618 3149 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:02:01.746608 3149 webserver.cc:492] Webserver started at http://127.1.157.195:39605/ using document root <none> and password file <none>
I20251214 03:02:01.747119 3149 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:01.747169 3149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:01.747351 3149 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:01.748950 3149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "5e046973a7df4072a4bc289adbf03ce0"
format_stamp: "Formatted at 2025-12-14 03:02:01 on dist-test-slave-k7z7"
server_key: "d5b31e736a9a328e3d76a70d8b7c9f8f"
server_key_iv: "97297d137d40083d55896db7cdcacd5e"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.749411 3149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "5e046973a7df4072a4bc289adbf03ce0"
format_stamp: "Formatted at 2025-12-14 03:02:01 on dist-test-slave-k7z7"
server_key: "d5b31e736a9a328e3d76a70d8b7c9f8f"
server_key_iv: "97297d137d40083d55896db7cdcacd5e"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.752643 3149 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251214 03:02:01.754916 3164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:01.755897 3149 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:01.755996 3149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "5e046973a7df4072a4bc289adbf03ce0"
format_stamp: "Formatted at 2025-12-14 03:02:01 on dist-test-slave-k7z7"
server_key: "d5b31e736a9a328e3d76a70d8b7c9f8f"
server_key_iv: "97297d137d40083d55896db7cdcacd5e"
server_key_version: "encryptionkey@0"
I20251214 03:02:01.756076 3149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:01.772845 3149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:01.775709 3149 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:01.775876 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:01.777424 3149 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:01.777478 3149 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:01.777531 3149 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:01.777559 3149 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 14 03:02:01 dist-test-slave-k7z7 krb5kdc[2743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:01.788008 3149 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:41531
I20251214 03:02:01.788033 3279 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:41531 every 8 connection(s)
I20251214 03:02:01.789069 3149 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:02:01.789525 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.777989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:52995 (local address 127.1.157.254:33877)
1214 03:02:01.778168 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.778173 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.779198 (+ 1025us) server_negotiation.cc:408] Connection header received
1214 03:02:01.780181 (+ 983us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.780186 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.780248 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.780305 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:01.782169 (+ 1864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.782937 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.783676 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.783860 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.785898 (+ 2038us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:01.785921 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:01.785925 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:01.785955 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:01.787529 (+ 1574us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.788095 (+ 566us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.788099 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.788101 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.788158 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.788444 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.788448 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.788450 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.788654 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:01.788760 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.789203 (+ 443us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.789348 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251214 03:02:01.790084 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.778271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33877 (local address 127.1.157.195:52995)
1214 03:02:01.778808 (+ 537us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:01.778872 (+ 64us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:01.779940 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:01.780711 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:01.780721 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:01.781271 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:01.782023 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.782040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.783085 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.783089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:01.783553 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.783566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.783782 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.784377 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:01.784408 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:01.785754 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:01.787664 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.787670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.787681 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.787996 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.788266 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.788269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.788270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.788357 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.788747 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:01.788752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:01.788952 (+ 200us) client_negotiation.cc:769] Sending connection context
1214 03:02:01.789104 (+ 152us) client_negotiation.cc:240] Negotiation successful
1214 03:02:01.789359 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20251214 03:02:01.791148 3280 heartbeater.cc:344] Connected to a master server at 127.1.157.254:33877
I20251214 03:02:01.791319 3280 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:01.791821 3280 heartbeater.cc:507] Master 127.1.157.254:33877 requested a full tablet report, sending...
I20251214 03:02:01.792722 2790 ts_manager.cc:194] Registered new tserver with Master: 5e046973a7df4072a4bc289adbf03ce0 (127.1.157.195:41531)
I20251214 03:02:01.793394 2790 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:52995
I20251214 03:02:01.797310 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 3149
I20251214 03:02:01.797415 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:02:01.797649 1655 external_mini_cluster.cc:1468] Setting key ff99345940b018a4175c8d27a156b5a5
I20251214 03:02:01.798767 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.792983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33877 (local address 127.1.157.195:55357)
1214 03:02:01.793252 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:01.793264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:01.793350 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:01.793600 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:01.793603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:01.793780 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:01.794012 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.794019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.794904 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.794907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:01.795278 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.795284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.795364 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.795804 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:01.795816 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:01.796057 (+ 241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:01.797402 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.797405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.797408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.797565 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.797866 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:01.797869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:01.797872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:01.797930 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:01.798314 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:01.798317 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:01.798404 (+ 87us) client_negotiation.cc:769] Sending connection context
1214 03:02:01.798503 (+ 99us) client_negotiation.cc:240] Negotiation successful
1214 03:02:01.798596 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20251214 03:02:01.798794 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.792961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:55357 (local address 127.1.157.254:33877)
1214 03:02:01.793064 (+ 103us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.793067 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.793319 (+ 252us) server_negotiation.cc:408] Connection header received
1214 03:02:01.793427 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.793430 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.793471 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.793546 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:01.794116 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.794818 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.795366 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.795495 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.796141 (+ 646us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:01.796154 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:01.796156 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:01.796170 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:01.797315 (+ 1145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.797645 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.797649 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.797651 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.797749 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.798024 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.798028 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.798031 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.798207 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:01.798301 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.798572 (+ 271us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.798666 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251214 03:02:01.799546 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 14 03:02:01 dist-test-slave-k7z7 krb5kdc[2743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 14 03:02:01 dist-test-slave-k7z7 krb5kdc[2743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:01.833994 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.826065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.1.157.254:33877)
1214 03:02:01.826247 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.826253 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.826272 (+ 19us) server_negotiation.cc:408] Connection header received
1214 03:02:01.826337 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.826342 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.826407 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.826526 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:01.827682 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.828245 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.828955 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.829122 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.831239 (+ 2117us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:01.831261 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:01.831263 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:01.831290 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:01.832595 (+ 1305us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.832962 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.832965 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.832966 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.833006 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:01.833218 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:01.833221 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:01.833222 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:01.833395 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:01.833508 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.833725 (+ 217us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.833829 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20251214 03:02:01.836638 2791 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:02:01.839056 2791 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:01.851226 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.846850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:36031 (local address 127.0.0.1:43970)
1214 03:02:01.848156 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:01.848168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:01.848240 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:01.848693 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:01.848696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:01.848710 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:01.848919 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.848925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.849885 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.849889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:01.850759 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.850769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.850865 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.850915 (+ 50us) client_negotiation.cc:769] Sending connection context
1214 03:02:01.850987 (+ 72us) client_negotiation.cc:240] Negotiation successful
1214 03:02:01.851085 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":60,"threads_started":1}
I20251214 03:02:01.851639 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.847073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:34795 (local address 127.0.0.1:36650)
1214 03:02:01.847717 (+ 644us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:01.847759 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:01.847914 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:01.848765 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:01.848768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:01.848786 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:01.849005 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.849013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.850429 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.850432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:01.851235 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.851247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.851346 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.851363 (+ 17us) client_negotiation.cc:769] Sending connection context
1214 03:02:01.851419 (+ 56us) client_negotiation.cc:240] Negotiation successful
1214 03:02:01.851512 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"spinlock_wait_cycles":4736,"thread_start_us":78,"threads_started":1}
I20251214 03:02:01.852185 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.846974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.1.157.194:36031)
1214 03:02:01.848183 (+ 1209us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.848190 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.848213 (+ 23us) server_negotiation.cc:408] Connection header received
1214 03:02:01.848383 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.848389 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.848563 (+ 174us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.848703 (+ 140us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:01.849031 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.849785 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.851268 (+ 1483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.851788 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.851905 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.851964 (+ 59us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.852047 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":117,"threads_started":1}
I20251214 03:02:01.852324 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.847629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.1.157.195:41531)
1214 03:02:01.847937 (+ 308us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.847944 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.847970 (+ 26us) server_negotiation.cc:408] Connection header received
1214 03:02:01.848041 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.848045 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.848210 (+ 165us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.848345 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:01.848950 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.850071 (+ 1121us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.851355 (+ 1284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.852008 (+ 653us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.852084 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.852139 (+ 55us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.852197 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
I20251214 03:02:01.852506 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.847335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41531 (local address 127.0.0.1:46584)
1214 03:02:01.847735 (+ 400us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:01.847759 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:01.847913 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:01.848535 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:01.848540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:01.848559 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:01.848833 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.848840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.850186 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.850189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:01.851049 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:01.851057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.851144 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.851158 (+ 14us) client_negotiation.cc:769] Sending connection context
1214 03:02:01.851197 (+ 39us) client_negotiation.cc:240] Negotiation successful
1214 03:02:01.851243 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"spinlock_wait_cycles":12928,"thread_start_us":95,"threads_started":1}
I20251214 03:02:01.852774 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:01.847464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36650 (local address 127.1.157.193:34795)
1214 03:02:01.848409 (+ 945us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:01.848415 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:01.848434 (+ 19us) server_negotiation.cc:408] Connection header received
1214 03:02:01.848501 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:01.848506 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:01.848630 (+ 124us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:01.849141 (+ 511us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:01.849292 (+ 151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.850306 (+ 1014us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:01.851927 (+ 1621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:01.852427 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:01.852527 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:01.852581 (+ 54us) server_negotiation.cc:300] Negotiation successful
1214 03:02:01.852658 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":74,"threads_started":1}
I20251214 03:02:01.854984 3072 tablet_service.cc:1505] Processing CreateTablet for tablet b288e2d9990e4f2083854a8ba79deee9 (DEFAULT_TABLE table=test-table [id=fcb67ad0248c432b8d1d240b7e3e8e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:01.855054 3214 tablet_service.cc:1505] Processing CreateTablet for tablet b288e2d9990e4f2083854a8ba79deee9 (DEFAULT_TABLE table=test-table [id=fcb67ad0248c432b8d1d240b7e3e8e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:01.855052 2930 tablet_service.cc:1505] Processing CreateTablet for tablet b288e2d9990e4f2083854a8ba79deee9 (DEFAULT_TABLE table=test-table [id=fcb67ad0248c432b8d1d240b7e3e8e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:01.855836 3072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b288e2d9990e4f2083854a8ba79deee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:01.855894 2930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b288e2d9990e4f2083854a8ba79deee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:01.855893 3214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b288e2d9990e4f2083854a8ba79deee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:01.861070 3303 tablet_bootstrap.cc:492] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: Bootstrap starting.
I20251214 03:02:01.862226 3305 tablet_bootstrap.cc:492] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: Bootstrap starting.
I20251214 03:02:01.863574 3303 tablet_bootstrap.cc:654] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:01.864245 3305 tablet_bootstrap.cc:654] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:01.864833 3303 log.cc:826] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:01.865397 3305 log.cc:826] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:01.865468 3304 tablet_bootstrap.cc:492] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: Bootstrap starting.
I20251214 03:02:01.866642 3303 tablet_bootstrap.cc:492] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: No bootstrap required, opened a new log
I20251214 03:02:01.866832 3303 ts_tablet_manager.cc:1403] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20251214 03:02:01.867067 3305 tablet_bootstrap.cc:492] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: No bootstrap required, opened a new log
I20251214 03:02:01.867125 3304 tablet_bootstrap.cc:654] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:01.867233 3305 ts_tablet_manager.cc:1403] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251214 03:02:01.867846 3304 log.cc:826] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:01.870100 3303 raft_consensus.cc:359] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:01.870100 3305 raft_consensus.cc:359] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:01.870317 3305 raft_consensus.cc:385] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:01.870317 3303 raft_consensus.cc:385] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:01.870369 3304 tablet_bootstrap.cc:492] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: No bootstrap required, opened a new log
I20251214 03:02:01.870384 3303 raft_consensus.cc:740] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 175c4087c31945aa95ce73d2f507fe4f, State: Initialized, Role: FOLLOWER
I20251214 03:02:01.870401 3305 raft_consensus.cc:740] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e046973a7df4072a4bc289adbf03ce0, State: Initialized, Role: FOLLOWER
I20251214 03:02:01.870533 3304 ts_tablet_manager.cc:1403] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251214 03:02:01.870831 3303 consensus_queue.cc:260] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:01.870831 3305 consensus_queue.cc:260] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:01.871680 3138 heartbeater.cc:499] Master 127.1.157.254:33877 was elected leader, sending a full tablet report...
I20251214 03:02:01.871696 3280 heartbeater.cc:499] Master 127.1.157.254:33877 was elected leader, sending a full tablet report...
I20251214 03:02:01.871814 3305 ts_tablet_manager.cc:1434] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251214 03:02:01.871678 3303 ts_tablet_manager.cc:1434] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20251214 03:02:01.873135 3304 raft_consensus.cc:359] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:01.873324 3304 raft_consensus.cc:385] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:01.873368 3304 raft_consensus.cc:740] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9b7c7511984e87afd03e4bfbfc6c95, State: Initialized, Role: FOLLOWER
I20251214 03:02:01.873683 3304 consensus_queue.cc:260] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:01.874387 3304 ts_tablet_manager.cc:1434] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251214 03:02:01.874534 2996 heartbeater.cc:499] Master 127.1.157.254:33877 was elected leader, sending a full tablet report...
W20251214 03:02:02.040776 3281 tablet.cc:2378] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 03:02:02.045302 3310 raft_consensus.cc:493] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:02.045501 3310 raft_consensus.cc:515] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.046578 3310 leader_election.cc:290] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf9b7c7511984e87afd03e4bfbfc6c95 (127.1.157.193:34795), 5e046973a7df4072a4bc289adbf03ce0 (127.1.157.195:41531)
I20251214 03:02:02.049899 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.046856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41531 (local address 127.1.157.194:37053)
1214 03:02:02.046971 (+ 115us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:02.046986 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:02.047084 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:02.047371 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:02.047375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:02.047400 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:02.047640 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.047649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.048774 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.048778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:02.049570 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.049579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.049689 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.049706 (+ 17us) client_negotiation.cc:769] Sending connection context
1214 03:02:02.049741 (+ 35us) client_negotiation.cc:240] Negotiation successful
1214 03:02:02.049791 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20251214 03:02:02.050424 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.047012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:37053 (local address 127.1.157.195:41531)
1214 03:02:02.047138 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.047141 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.047157 (+ 16us) server_negotiation.cc:408] Connection header received
1214 03:02:02.047205 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.047207 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.047250 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.047337 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:02.047841 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.048635 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.049697 (+ 1062us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.050154 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.050195 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.050245 (+ 50us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.050305 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251214 03:02:02.050962 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.046856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:34795 (local address 127.1.157.194:60471)
1214 03:02:02.047500 (+ 644us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:02.047512 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:02.047593 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:02.047846 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:02.047849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:02.047871 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:02.048199 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.048207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.049120 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.049123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:02.049899 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.049908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.050670 (+ 762us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.050688 (+ 18us) client_negotiation.cc:769] Sending connection context
1214 03:02:02.050763 (+ 75us) client_negotiation.cc:240] Negotiation successful
1214 03:02:02.050826 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":327,"thread_start_us":93,"threads_started":1}
I20251214 03:02:02.050983 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.046946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:60471 (local address 127.1.157.193:34795)
1214 03:02:02.047101 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.047106 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.047588 (+ 482us) server_negotiation.cc:408] Connection header received
1214 03:02:02.047670 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.047674 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.047728 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.047795 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:02.048358 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.048948 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.050051 (+ 1103us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.050529 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.050590 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.050804 (+ 214us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.050877 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251214 03:02:02.051079 3234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b288e2d9990e4f2083854a8ba79deee9" candidate_uuid: "175c4087c31945aa95ce73d2f507fe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e046973a7df4072a4bc289adbf03ce0" is_pre_election: true
I20251214 03:02:02.051399 3234 raft_consensus.cc:2468] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 175c4087c31945aa95ce73d2f507fe4f in term 0.
I20251214 03:02:02.051926 3026 leader_election.cc:304] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 175c4087c31945aa95ce73d2f507fe4f, 5e046973a7df4072a4bc289adbf03ce0; no voters:
I20251214 03:02:02.052089 2950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b288e2d9990e4f2083854a8ba79deee9" candidate_uuid: "175c4087c31945aa95ce73d2f507fe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" is_pre_election: true
I20251214 03:02:02.052225 3310 raft_consensus.cc:2804] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:02.052314 3310 raft_consensus.cc:493] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:02.052341 2950 raft_consensus.cc:2468] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 175c4087c31945aa95ce73d2f507fe4f in term 0.
I20251214 03:02:02.052369 3310 raft_consensus.cc:3060] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:02.053422 3310 raft_consensus.cc:515] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.053782 3310 leader_election.cc:290] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [CANDIDATE]: Term 1 election: Requested vote from peers cf9b7c7511984e87afd03e4bfbfc6c95 (127.1.157.193:34795), 5e046973a7df4072a4bc289adbf03ce0 (127.1.157.195:41531)
I20251214 03:02:02.054131 3234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b288e2d9990e4f2083854a8ba79deee9" candidate_uuid: "175c4087c31945aa95ce73d2f507fe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e046973a7df4072a4bc289adbf03ce0"
I20251214 03:02:02.054131 2950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b288e2d9990e4f2083854a8ba79deee9" candidate_uuid: "175c4087c31945aa95ce73d2f507fe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95"
I20251214 03:02:02.054252 3234 raft_consensus.cc:3060] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:02.054270 2950 raft_consensus.cc:3060] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:02.055413 3234 raft_consensus.cc:2468] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 175c4087c31945aa95ce73d2f507fe4f in term 1.
I20251214 03:02:02.055413 2950 raft_consensus.cc:2468] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 175c4087c31945aa95ce73d2f507fe4f in term 1.
I20251214 03:02:02.055776 3026 leader_election.cc:304] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 175c4087c31945aa95ce73d2f507fe4f, 5e046973a7df4072a4bc289adbf03ce0, cf9b7c7511984e87afd03e4bfbfc6c95; no voters:
I20251214 03:02:02.055958 3310 raft_consensus.cc:2804] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:02.056231 3310 raft_consensus.cc:697] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 1 LEADER]: Becoming Leader. State: Replica: 175c4087c31945aa95ce73d2f507fe4f, State: Running, Role: LEADER
I20251214 03:02:02.056550 3310 consensus_queue.cc:237] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.059684 2790 catalog_manager.cc:5654] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 175c4087c31945aa95ce73d2f507fe4f (127.1.157.194). New cstate: current_term: 1 leader_uuid: "175c4087c31945aa95ce73d2f507fe4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } health_report { overall_health: UNKNOWN } } }
W20251214 03:02:02.068166 3139 tablet.cc:2378] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251214 03:02:02.083688 2997 tablet.cc:2378] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 03:02:02.087729 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.084680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.1.157.194:36031)
1214 03:02:02.084814 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.084819 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.084833 (+ 14us) server_negotiation.cc:408] Connection header received
1214 03:02:02.084968 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.084971 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.085026 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.085100 (+ 74us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:02.085494 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.085954 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.086660 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.086867 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.086920 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:02.087340 (+ 420us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:02.087440 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.087532 (+ 92us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.087607 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251214 03:02:02.092533 2950 raft_consensus.cc:1275] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 FOLLOWER]: Refusing update from remote peer 175c4087c31945aa95ce73d2f507fe4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:02.092533 3234 raft_consensus.cc:1275] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0 [term 1 FOLLOWER]: Refusing update from remote peer 175c4087c31945aa95ce73d2f507fe4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:02.093092 3310 consensus_queue.cc:1048] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:02.093302 3313 consensus_queue.cc:1048] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:02.100548 3317 mvcc.cc:204] Tried to move back new op lower bound from 7232230695285043200 to 7232230695146758144. Current Snapshot: MvccSnapshot[applied={T|T < 7232230695285043200}]
I20251214 03:02:02.110136 2790 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44876:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:02.110299 2790 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44876:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:02.112624 2790 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0: Sending DeleteTablet for 3 replicas of tablet b288e2d9990e4f2083854a8ba79deee9
I20251214 03:02:02.113359 2930 tablet_service.cc:1552] Processing DeleteTablet for tablet b288e2d9990e4f2083854a8ba79deee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:02 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:36650
I20251214 03:02:02.113358 3072 tablet_service.cc:1552] Processing DeleteTablet for tablet b288e2d9990e4f2083854a8ba79deee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:02 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:43970
I20251214 03:02:02.114018 3327 tablet_replica.cc:333] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: stopping tablet replica
I20251214 03:02:02.114271 3326 tablet_replica.cc:333] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: stopping tablet replica
I20251214 03:02:02.114212 3214 tablet_service.cc:1552] Processing DeleteTablet for tablet b288e2d9990e4f2083854a8ba79deee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:02 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:46584
I20251214 03:02:02.114216 2790 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:02:02.114653 2790 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:02.114671 3328 tablet_replica.cc:333] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: stopping tablet replica
I20251214 03:02:02.114730 3327 raft_consensus.cc:2243] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:02.114902 3328 raft_consensus.cc:2243] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:02.115097 3327 raft_consensus.cc:2272] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:02.115455 3328 raft_consensus.cc:2272] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:02.115523 3326 raft_consensus.cc:2243] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:02:02.115938 3326 raft_consensus.cc:2272] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:02.117027 3328 ts_tablet_manager.cc:1916] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:02.117118 3326 ts_tablet_manager.cc:1916] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:02.118140 3327 ts_tablet_manager.cc:1916] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:02.119547 3328 ts_tablet_manager.cc:1929] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:02:02.119659 3328 log.cc:1199] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/b288e2d9990e4f2083854a8ba79deee9
I20251214 03:02:02.119640 3326 ts_tablet_manager.cc:1929] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:02:02.119719 3326 log.cc:1199] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/b288e2d9990e4f2083854a8ba79deee9
I20251214 03:02:02.119973 3328 ts_tablet_manager.cc:1950] T b288e2d9990e4f2083854a8ba79deee9 P 5e046973a7df4072a4bc289adbf03ce0: Deleting consensus metadata
I20251214 03:02:02.120023 3326 ts_tablet_manager.cc:1950] T b288e2d9990e4f2083854a8ba79deee9 P 175c4087c31945aa95ce73d2f507fe4f: Deleting consensus metadata
I20251214 03:02:02.120519 3327 ts_tablet_manager.cc:1929] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:02:02.120605 3327 log.cc:1199] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/wal/wals/b288e2d9990e4f2083854a8ba79deee9
I20251214 03:02:02.120875 2930 tablet_service.cc:1505] Processing CreateTablet for tablet 2c7821011a7c495fa8506733a7ac1edc (DEFAULT_TABLE table=test-table [id=046373f8318c4bb3bc317c365091b539]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:02.121147 2930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7821011a7c495fa8506733a7ac1edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:02.122720 3304 tablet_bootstrap.cc:492] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95: Bootstrap starting.
I20251214 03:02:02.122841 2777 catalog_manager.cc:4985] TS 175c4087c31945aa95ce73d2f507fe4f (127.1.157.194:36031): tablet b288e2d9990e4f2083854a8ba79deee9 (table test-table [id=fcb67ad0248c432b8d1d240b7e3e8e54]) successfully deleted
I20251214 03:02:02.123102 2778 catalog_manager.cc:4985] TS 5e046973a7df4072a4bc289adbf03ce0 (127.1.157.195:41531): tablet b288e2d9990e4f2083854a8ba79deee9 (table test-table [id=fcb67ad0248c432b8d1d240b7e3e8e54]) successfully deleted
I20251214 03:02:02.123421 3214 tablet_service.cc:1505] Processing CreateTablet for tablet 2c7821011a7c495fa8506733a7ac1edc (DEFAULT_TABLE table=test-table [id=046373f8318c4bb3bc317c365091b539]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:02.123656 3214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7821011a7c495fa8506733a7ac1edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:02.123649 3072 tablet_service.cc:1505] Processing CreateTablet for tablet 2c7821011a7c495fa8506733a7ac1edc (DEFAULT_TABLE table=test-table [id=046373f8318c4bb3bc317c365091b539]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:02.123744 3304 tablet_bootstrap.cc:654] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:02.123886 3072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c7821011a7c495fa8506733a7ac1edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:02.125077 3304 tablet_bootstrap.cc:492] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95: No bootstrap required, opened a new log
I20251214 03:02:02.125156 3304 ts_tablet_manager.cc:1403] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:02.125669 3304 raft_consensus.cc:359] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.125770 3304 raft_consensus.cc:385] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:02.125800 3304 raft_consensus.cc:740] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9b7c7511984e87afd03e4bfbfc6c95, State: Initialized, Role: FOLLOWER
I20251214 03:02:02.125821 3305 tablet_bootstrap.cc:492] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0: Bootstrap starting.
I20251214 03:02:02.125948 3303 tablet_bootstrap.cc:492] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f: Bootstrap starting.
I20251214 03:02:02.125926 3304 consensus_queue.cc:260] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.126173 3304 ts_tablet_manager.cc:1434] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:02.126803 3303 tablet_bootstrap.cc:654] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:02.126803 3305 tablet_bootstrap.cc:654] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:02.120894 3327 ts_tablet_manager.cc:1950] T b288e2d9990e4f2083854a8ba79deee9 P cf9b7c7511984e87afd03e4bfbfc6c95: Deleting consensus metadata
I20251214 03:02:02.128587 2776 catalog_manager.cc:4985] TS cf9b7c7511984e87afd03e4bfbfc6c95 (127.1.157.193:34795): tablet b288e2d9990e4f2083854a8ba79deee9 (table test-table [id=fcb67ad0248c432b8d1d240b7e3e8e54]) successfully deleted
I20251214 03:02:02.129308 3305 tablet_bootstrap.cc:492] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0: No bootstrap required, opened a new log
I20251214 03:02:02.129400 3305 ts_tablet_manager.cc:1403] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251214 03:02:02.129607 3303 tablet_bootstrap.cc:492] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f: No bootstrap required, opened a new log
I20251214 03:02:02.129689 3303 ts_tablet_manager.cc:1403] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20251214 03:02:02.129886 3305 raft_consensus.cc:359] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.129993 3305 raft_consensus.cc:385] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:02.130033 3305 raft_consensus.cc:740] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e046973a7df4072a4bc289adbf03ce0, State: Initialized, Role: FOLLOWER
I20251214 03:02:02.130146 3303 raft_consensus.cc:359] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.130162 3305 consensus_queue.cc:260] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.130241 3303 raft_consensus.cc:385] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:02.130277 3303 raft_consensus.cc:740] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 175c4087c31945aa95ce73d2f507fe4f, State: Initialized, Role: FOLLOWER
I20251214 03:02:02.130388 3303 consensus_queue.cc:260] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.130517 3305 ts_tablet_manager.cc:1434] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251214 03:02:02.130656 3303 ts_tablet_manager.cc:1434] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251214 03:02:02.257436 3311 raft_consensus.cc:493] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:02.257624 3311 raft_consensus.cc:515] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.259004 3311 leader_election.cc:290] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 175c4087c31945aa95ce73d2f507fe4f (127.1.157.194:36031), 5e046973a7df4072a4bc289adbf03ce0 (127.1.157.195:41531)
I20251214 03:02:02.262478 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.259216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41531 (local address 127.1.157.193:58111)
1214 03:02:02.259576 (+ 360us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:02.259592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:02.259701 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:02.260088 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:02.260092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:02.260113 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:02.260419 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.260428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.261126 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.261130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:02.262039 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.262049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.262159 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.262190 (+ 31us) client_negotiation.cc:769] Sending connection context
1214 03:02:02.262241 (+ 51us) client_negotiation.cc:240] Negotiation successful
1214 03:02:02.262308 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":94,"threads_started":1}
I20251214 03:02:02.262853 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.259525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:58111 (local address 127.1.157.195:41531)
1214 03:02:02.259676 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.259680 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.259696 (+ 16us) server_negotiation.cc:408] Connection header received
1214 03:02:02.259861 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.259865 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.259921 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.260018 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:02.260539 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.261013 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.262156 (+ 1143us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.262601 (+ 445us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.262643 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.262697 (+ 54us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.262756 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251214 03:02:02.263301 3234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c7821011a7c495fa8506733a7ac1edc" candidate_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e046973a7df4072a4bc289adbf03ce0" is_pre_election: true
I20251214 03:02:02.263459 3234 raft_consensus.cc:2468] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf9b7c7511984e87afd03e4bfbfc6c95 in term 0.
I20251214 03:02:02.263674 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.259239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:36031 (local address 127.1.157.193:34089)
1214 03:02:02.259682 (+ 443us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:02.259692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:02.259817 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:02.260160 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:02.260164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:02.260186 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:02.261043 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.261064 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.262059 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.262063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:02.263243 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.263254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.263376 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.263393 (+ 17us) client_negotiation.cc:769] Sending connection context
1214 03:02:02.263444 (+ 51us) client_negotiation.cc:240] Negotiation successful
1214 03:02:02.263515 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":96,"threads_started":1}
I20251214 03:02:02.264107 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.259407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:34089 (local address 127.1.157.194:36031)
1214 03:02:02.259564 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.259568 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.259755 (+ 187us) server_negotiation.cc:408] Connection header received
1214 03:02:02.259927 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.259930 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.259976 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.260040 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:02.261204 (+ 1164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.261926 (+ 722us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.263374 (+ 1448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.263860 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.263900 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.263953 (+ 53us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.264015 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251214 03:02:02.264079 2884 leader_election.cc:304] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e046973a7df4072a4bc289adbf03ce0, cf9b7c7511984e87afd03e4bfbfc6c95; no voters:
I20251214 03:02:02.264351 3311 raft_consensus.cc:2804] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:02.264420 3311 raft_consensus.cc:493] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:02.264467 3311 raft_consensus.cc:3060] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:02.264590 3092 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c7821011a7c495fa8506733a7ac1edc" candidate_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "175c4087c31945aa95ce73d2f507fe4f" is_pre_election: true
I20251214 03:02:02.264793 3092 raft_consensus.cc:2468] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf9b7c7511984e87afd03e4bfbfc6c95 in term 0.
I20251214 03:02:02.265710 3311 raft_consensus.cc:515] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.266104 3311 leader_election.cc:290] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [CANDIDATE]: Term 1 election: Requested vote from peers 175c4087c31945aa95ce73d2f507fe4f (127.1.157.194:36031), 5e046973a7df4072a4bc289adbf03ce0 (127.1.157.195:41531)
I20251214 03:02:02.266475 3234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c7821011a7c495fa8506733a7ac1edc" candidate_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e046973a7df4072a4bc289adbf03ce0"
I20251214 03:02:02.266490 3092 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c7821011a7c495fa8506733a7ac1edc" candidate_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "175c4087c31945aa95ce73d2f507fe4f"
I20251214 03:02:02.266592 3092 raft_consensus.cc:3060] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:02.266593 3234 raft_consensus.cc:3060] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:02.267407 3234 raft_consensus.cc:2468] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf9b7c7511984e87afd03e4bfbfc6c95 in term 1.
I20251214 03:02:02.267625 3092 raft_consensus.cc:2468] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf9b7c7511984e87afd03e4bfbfc6c95 in term 1.
I20251214 03:02:02.267695 2884 leader_election.cc:304] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e046973a7df4072a4bc289adbf03ce0, cf9b7c7511984e87afd03e4bfbfc6c95; no voters:
I20251214 03:02:02.267879 3311 raft_consensus.cc:2804] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:02.268070 3311 raft_consensus.cc:697] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 LEADER]: Becoming Leader. State: Replica: cf9b7c7511984e87afd03e4bfbfc6c95, State: Running, Role: LEADER
I20251214 03:02:02.268308 3311 consensus_queue.cc:237] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.271188 2791 catalog_manager.cc:5654] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf9b7c7511984e87afd03e4bfbfc6c95 (127.1.157.193). New cstate: current_term: 1 leader_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } health_report { overall_health: UNKNOWN } } }
Dec 14 03:02:02 dist-test-slave-k7z7 krb5kdc[2743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681320, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:02.309460 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.302770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33877 (local address 127.0.0.1:44892)
1214 03:02:02.303009 (+ 239us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:02.303024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:02.303121 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:02.303385 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:02.303387 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:02.303773 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:02.303961 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.303967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.304660 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.304663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:02.305041 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.305047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.305125 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.305677 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:02.305690 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:02.306949 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:02.308307 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:02.308314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:02.308318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:02.308505 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:02.308662 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:02.308664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:02.308665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:02.308707 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:02.308977 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:02.308982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:02.309073 (+ 91us) client_negotiation.cc:769] Sending connection context
1214 03:02:02.309160 (+ 87us) client_negotiation.cc:240] Negotiation successful
1214 03:02:02.309302 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20251214 03:02:02.309476 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.302819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.1.157.254:33877)
1214 03:02:02.302938 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.302941 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.303088 (+ 147us) server_negotiation.cc:408] Connection header received
1214 03:02:02.303240 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.303243 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.303288 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.303349 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:02.304061 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.304572 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.305120 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.305303 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.307043 (+ 1740us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:02.307056 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:02.307058 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:02.307078 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:02.308229 (+ 1151us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:02.308571 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:02.308573 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:02.308574 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:02.308608 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:02.308768 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:02.308770 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:02.308771 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:02.308892 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:02.308980 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.309259 (+ 279us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.309347 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251214 03:02:02.312997 2791 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:44892:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251214 03:02:02.313735 2791 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:02.318115 2930 tablet_service.cc:1505] Processing CreateTablet for tablet 62a674dcdb2e4c26ad24a97876700048 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97f90e85600f4806b3afa08d4cd2aba5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251214 03:02:02.318282 3072 tablet_service.cc:1505] Processing CreateTablet for tablet 62a674dcdb2e4c26ad24a97876700048 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97f90e85600f4806b3afa08d4cd2aba5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251214 03:02:02.318492 2930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62a674dcdb2e4c26ad24a97876700048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:02.318425 3214 tablet_service.cc:1505] Processing CreateTablet for tablet 62a674dcdb2e4c26ad24a97876700048 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97f90e85600f4806b3afa08d4cd2aba5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251214 03:02:02.318611 3072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62a674dcdb2e4c26ad24a97876700048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:02.318730 3214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62a674dcdb2e4c26ad24a97876700048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:02.320586 3303 tablet_bootstrap.cc:492] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: Bootstrap starting.
I20251214 03:02:02.321112 3305 tablet_bootstrap.cc:492] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: Bootstrap starting.
I20251214 03:02:02.321321 3304 tablet_bootstrap.cc:492] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: Bootstrap starting.
I20251214 03:02:02.321497 3303 tablet_bootstrap.cc:654] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:02.321914 3305 tablet_bootstrap.cc:654] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:02.322186 3304 tablet_bootstrap.cc:654] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:02.322589 3303 tablet_bootstrap.cc:492] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: No bootstrap required, opened a new log
I20251214 03:02:02.322662 3303 ts_tablet_manager.cc:1403] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:02.323076 3303 raft_consensus.cc:359] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.323173 3303 raft_consensus.cc:385] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:02.323202 3303 raft_consensus.cc:740] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 175c4087c31945aa95ce73d2f507fe4f, State: Initialized, Role: FOLLOWER
I20251214 03:02:02.323319 3303 consensus_queue.cc:260] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.323355 3305 tablet_bootstrap.cc:492] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: No bootstrap required, opened a new log
I20251214 03:02:02.323446 3305 ts_tablet_manager.cc:1403] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251214 03:02:02.323578 3304 tablet_bootstrap.cc:492] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: No bootstrap required, opened a new log
I20251214 03:02:02.323607 3303 ts_tablet_manager.cc:1434] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:02.323663 3304 ts_tablet_manager.cc:1403] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:02.323951 3305 raft_consensus.cc:359] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.324062 3305 raft_consensus.cc:385] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:02.324097 3305 raft_consensus.cc:740] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e046973a7df4072a4bc289adbf03ce0, State: Initialized, Role: FOLLOWER
I20251214 03:02:02.324132 3304 raft_consensus.cc:359] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.324225 3304 raft_consensus.cc:385] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:02.324258 3304 raft_consensus.cc:740] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9b7c7511984e87afd03e4bfbfc6c95, State: Initialized, Role: FOLLOWER
I20251214 03:02:02.324234 3305 consensus_queue.cc:260] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.324297 3313 tablet_replica.cc:442] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } } }
I20251214 03:02:02.324390 3313 tablet_replica.cc:445] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:02.324384 3304 consensus_queue.cc:260] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.324492 3309 tablet_replica.cc:442] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } } }
I20251214 03:02:02.324506 3305 ts_tablet_manager.cc:1434] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:02.324625 3309 tablet_replica.cc:445] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:02.324640 3304 ts_tablet_manager.cc:1434] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:02.324967 3311 tablet_replica.cc:442] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } } }
I20251214 03:02:02.325078 3311 tablet_replica.cc:445] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:02.330160 3313 raft_consensus.cc:493] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:02.330266 3313 raft_consensus.cc:515] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.330770 3313 leader_election.cc:290] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf9b7c7511984e87afd03e4bfbfc6c95 (127.1.157.193:34795), 5e046973a7df4072a4bc289adbf03ce0 (127.1.157.195:41531)
I20251214 03:02:02.331162 2950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62a674dcdb2e4c26ad24a97876700048" candidate_uuid: "175c4087c31945aa95ce73d2f507fe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" is_pre_election: true
I20251214 03:02:02.331252 3234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62a674dcdb2e4c26ad24a97876700048" candidate_uuid: "175c4087c31945aa95ce73d2f507fe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e046973a7df4072a4bc289adbf03ce0" is_pre_election: true
I20251214 03:02:02.331301 2950 raft_consensus.cc:2468] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 175c4087c31945aa95ce73d2f507fe4f in term 0.
I20251214 03:02:02.331403 3234 raft_consensus.cc:2468] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 175c4087c31945aa95ce73d2f507fe4f in term 0.
I20251214 03:02:02.331701 3024 leader_election.cc:304] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 175c4087c31945aa95ce73d2f507fe4f, cf9b7c7511984e87afd03e4bfbfc6c95; no voters:
I20251214 03:02:02.331890 3313 raft_consensus.cc:2804] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:02.331943 3313 raft_consensus.cc:493] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:02.331996 3313 raft_consensus.cc:3060] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:02.332959 3313 raft_consensus.cc:515] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.333369 3313 leader_election.cc:290] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [CANDIDATE]: Term 1 election: Requested vote from peers cf9b7c7511984e87afd03e4bfbfc6c95 (127.1.157.193:34795), 5e046973a7df4072a4bc289adbf03ce0 (127.1.157.195:41531)
I20251214 03:02:02.333658 2950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62a674dcdb2e4c26ad24a97876700048" candidate_uuid: "175c4087c31945aa95ce73d2f507fe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95"
I20251214 03:02:02.333690 3234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62a674dcdb2e4c26ad24a97876700048" candidate_uuid: "175c4087c31945aa95ce73d2f507fe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e046973a7df4072a4bc289adbf03ce0"
I20251214 03:02:02.333782 3234 raft_consensus.cc:3060] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:02.333782 2950 raft_consensus.cc:3060] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:02.334806 3234 raft_consensus.cc:2468] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 175c4087c31945aa95ce73d2f507fe4f in term 1.
I20251214 03:02:02.334806 2950 raft_consensus.cc:2468] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 175c4087c31945aa95ce73d2f507fe4f in term 1.
I20251214 03:02:02.335155 3024 leader_election.cc:304] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 175c4087c31945aa95ce73d2f507fe4f, cf9b7c7511984e87afd03e4bfbfc6c95; no voters:
I20251214 03:02:02.335366 3313 raft_consensus.cc:2804] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:02.335450 3313 raft_consensus.cc:697] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [term 1 LEADER]: Becoming Leader. State: Replica: 175c4087c31945aa95ce73d2f507fe4f, State: Running, Role: LEADER
I20251214 03:02:02.335597 3313 consensus_queue.cc:237] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } }
I20251214 03:02:02.336457 3310 tablet_replica.cc:442] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: TxnStatusTablet state changed. Reason: New leader 175c4087c31945aa95ce73d2f507fe4f. Latest consensus state: current_term: 1 leader_uuid: "175c4087c31945aa95ce73d2f507fe4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } } }
I20251214 03:02:02.336560 3310 tablet_replica.cc:445] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: This TxnStatusTablet replica's current role is: LEADER
I20251214 03:02:02.336809 3346 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251214 03:02:02.336993 3346 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251214 03:02:02.337532 2790 catalog_manager.cc:5654] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 175c4087c31945aa95ce73d2f507fe4f (127.1.157.194). New cstate: current_term: 1 leader_uuid: "175c4087c31945aa95ce73d2f507fe4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } health_report { overall_health: UNKNOWN } } }
I20251214 03:02:02.348407 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.345719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.1.157.194:36031)
1214 03:02:02.345842 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.345846 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.345860 (+ 14us) server_negotiation.cc:408] Connection header received
1214 03:02:02.346034 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.346036 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.346089 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.346185 (+ 96us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:02.346563 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.347035 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.347711 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.347897 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.347941 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:02.348017 (+ 76us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:02.348087 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.348205 (+ 118us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.348260 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251214 03:02:02.348407 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.345629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:36031 (local address 127.0.0.1:43988)
1214 03:02:02.345785 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:02.345803 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:02.345923 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:02.346229 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:02.346232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:02.346241 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:02:02.346444 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.346450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.347147 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.347151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:02.347601 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.347609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.347717 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.347768 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:02:02.348103 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:02:02.348122 (+ 19us) client_negotiation.cc:769] Sending connection context
1214 03:02:02.348187 (+ 65us) client_negotiation.cc:240] Negotiation successful
1214 03:02:02.348268 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20251214 03:02:02.351181 3234 raft_consensus.cc:1275] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0 [term 1 FOLLOWER]: Refusing update from remote peer 175c4087c31945aa95ce73d2f507fe4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:02.351181 2950 raft_consensus.cc:1275] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 FOLLOWER]: Refusing update from remote peer 175c4087c31945aa95ce73d2f507fe4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:02.351600 3313 consensus_queue.cc:1048] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:02.351907 3310 consensus_queue.cc:1048] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:02.353260 3309 tablet_replica.cc:442] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: TxnStatusTablet state changed. Reason: New leader 175c4087c31945aa95ce73d2f507fe4f. Latest consensus state: current_term: 1 leader_uuid: "175c4087c31945aa95ce73d2f507fe4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } } }
I20251214 03:02:02.353374 3309 tablet_replica.cc:445] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:02.353739 3311 tablet_replica.cc:442] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: TxnStatusTablet state changed. Reason: New leader 175c4087c31945aa95ce73d2f507fe4f. Latest consensus state: current_term: 1 leader_uuid: "175c4087c31945aa95ce73d2f507fe4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } } }
I20251214 03:02:02.353839 3311 tablet_replica.cc:445] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:02.354799 3321 tablet_replica.cc:442] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "175c4087c31945aa95ce73d2f507fe4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } } }
I20251214 03:02:02.354898 3321 tablet_replica.cc:445] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: This TxnStatusTablet replica's current role is: LEADER
I20251214 03:02:02.354979 3316 mvcc.cc:204] Tried to move back new op lower bound from 7232230696346374144 to 7232230696288575488. Current Snapshot: MvccSnapshot[applied={T|T < 7232230696346374144}]
I20251214 03:02:02.355808 3313 tablet_replica.cc:442] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "175c4087c31945aa95ce73d2f507fe4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } } }
I20251214 03:02:02.355895 3313 tablet_replica.cc:445] T 62a674dcdb2e4c26ad24a97876700048 P 175c4087c31945aa95ce73d2f507fe4f: This TxnStatusTablet replica's current role is: LEADER
I20251214 03:02:02.356635 3318 mvcc.cc:204] Tried to move back new op lower bound from 7232230696346374144 to 7232230696288575488. Current Snapshot: MvccSnapshot[applied={T|T < 7232230696346374144}]
I20251214 03:02:02.357393 3309 tablet_replica.cc:442] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "175c4087c31945aa95ce73d2f507fe4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } } }
I20251214 03:02:02.357489 3309 tablet_replica.cc:445] T 62a674dcdb2e4c26ad24a97876700048 P 5e046973a7df4072a4bc289adbf03ce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:02.358177 3311 tablet_replica.cc:442] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "175c4087c31945aa95ce73d2f507fe4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9b7c7511984e87afd03e4bfbfc6c95" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34795 } } peers { permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 } } peers { permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 } } }
I20251214 03:02:02.358274 3311 tablet_replica.cc:445] T 62a674dcdb2e4c26ad24a97876700048 P cf9b7c7511984e87afd03e4bfbfc6c95: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:02.363143 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.359820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.1.157.193:34795)
1214 03:02:02.359939 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.359944 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.359980 (+ 36us) server_negotiation.cc:408] Connection header received
1214 03:02:02.360054 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.360057 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.360137 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.360237 (+ 100us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:02.360544 (+ 307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.361314 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.361957 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.362153 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.362220 (+ 67us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:02.362759 (+ 539us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:02.362870 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.362964 (+ 94us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.363017 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251214 03:02:02.372474 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.369583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:36031 (local address 127.1.157.193:56689)
1214 03:02:02.369826 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:02.369838 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:02.369921 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:02.370133 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:02.370135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:02.370141 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:02:02.370353 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.370358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.371292 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.371294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:02.371710 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.371716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.371802 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.371832 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:02:02.372226 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:02:02.372231 (+ 5us) client_negotiation.cc:769] Sending connection context
1214 03:02:02.372302 (+ 71us) client_negotiation.cc:240] Negotiation successful
1214 03:02:02.372359 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20251214 03:02:02.372484 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.369679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:56689 (local address 127.1.157.194:36031)
1214 03:02:02.369806 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.369810 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.369895 (+ 85us) server_negotiation.cc:408] Connection header received
1214 03:02:02.369998 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.370001 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.370051 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.370129 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:02.370441 (+ 312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.371164 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.371800 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.371983 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.372025 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:02.372096 (+ 71us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:02.372173 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.372320 (+ 147us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.372365 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251214 03:02:02.378923 3234 raft_consensus.cc:1275] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0 [term 1 FOLLOWER]: Refusing update from remote peer cf9b7c7511984e87afd03e4bfbfc6c95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:02.379073 3092 raft_consensus.cc:1275] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f [term 1 FOLLOWER]: Refusing update from remote peer cf9b7c7511984e87afd03e4bfbfc6c95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:02.379523 3334 consensus_queue.cc:1048] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e046973a7df4072a4bc289adbf03ce0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:02.379676 3311 consensus_queue.cc:1048] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "175c4087c31945aa95ce73d2f507fe4f" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 36031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:02.401249 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.398075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:46705 (local address 127.1.157.193:34795)
1214 03:02:02.398189 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.398194 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.398479 (+ 285us) server_negotiation.cc:408] Connection header received
1214 03:02:02.398603 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.398606 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.398654 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.398719 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:02.399143 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.399660 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.400475 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.400688 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.400737 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:02.400820 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:02.400896 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.401030 (+ 134us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.401080 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251214 03:02:02.401240 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.397962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:34795 (local address 127.1.157.194:46705)
1214 03:02:02.398389 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:02.398403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:02.398517 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:02.398736 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:02.398739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:02.398746 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:02:02.399034 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.399041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.399775 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.399779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:02.400277 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:02.400288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.400476 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.400521 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:02:02.400937 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:02:02.400943 (+ 6us) client_negotiation.cc:769] Sending connection context
1214 03:02:02.401012 (+ 69us) client_negotiation.cc:240] Negotiation successful
1214 03:02:02.401080 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":104,"threads_started":1}
I20251214 03:02:02.426949 2791 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44876:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:02.427137 2791 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44876:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:02.429667 2791 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 169c7196f42b43dd8aa03cf57da7f9e0: Sending DeleteTablet for 3 replicas of tablet 2c7821011a7c495fa8506733a7ac1edc
I20251214 03:02:02.430401 2930 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c7821011a7c495fa8506733a7ac1edc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:02 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:36650
I20251214 03:02:02.430413 3214 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c7821011a7c495fa8506733a7ac1edc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:02 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:46584
I20251214 03:02:02.430606 3328 tablet_replica.cc:333] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0: stopping tablet replica
I20251214 03:02:02.430716 3328 raft_consensus.cc:2243] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:02.430827 3328 raft_consensus.cc:2272] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:02.430979 3327 tablet_replica.cc:333] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95: stopping tablet replica
I20251214 03:02:02.431097 3327 raft_consensus.cc:2243] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:02:02.431397 3327 raft_consensus.cc:2272] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:02.431432 3072 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c7821011a7c495fa8506733a7ac1edc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:02 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:43970
I20251214 03:02:02.431625 3326 tablet_replica.cc:333] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f: stopping tablet replica
I20251214 03:02:02.431730 3326 raft_consensus.cc:2243] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:02.431840 3326 raft_consensus.cc:2272] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:02.432338 3327 ts_tablet_manager.cc:1916] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:02.433940 3326 ts_tablet_manager.cc:1916] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:02.435250 3327 ts_tablet_manager.cc:1929] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:02:02.435328 3327 log.cc:1199] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-0/wal/wals/2c7821011a7c495fa8506733a7ac1edc
I20251214 03:02:02.435564 3327 ts_tablet_manager.cc:1950] T 2c7821011a7c495fa8506733a7ac1edc P cf9b7c7511984e87afd03e4bfbfc6c95: Deleting consensus metadata
I20251214 03:02:02.436369 3328 ts_tablet_manager.cc:1916] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:02.436462 3326 ts_tablet_manager.cc:1929] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:02:02.436540 3326 log.cc:1199] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/2c7821011a7c495fa8506733a7ac1edc
I20251214 03:02:02.436609 2776 catalog_manager.cc:4985] TS cf9b7c7511984e87afd03e4bfbfc6c95 (127.1.157.193:34795): tablet 2c7821011a7c495fa8506733a7ac1edc (table test-table [id=046373f8318c4bb3bc317c365091b539]) successfully deleted
I20251214 03:02:02.436758 3326 ts_tablet_manager.cc:1950] T 2c7821011a7c495fa8506733a7ac1edc P 175c4087c31945aa95ce73d2f507fe4f: Deleting consensus metadata
I20251214 03:02:02.437470 2777 catalog_manager.cc:4985] TS 175c4087c31945aa95ce73d2f507fe4f (127.1.157.194:36031): tablet 2c7821011a7c495fa8506733a7ac1edc (table test-table [id=046373f8318c4bb3bc317c365091b539]) successfully deleted
I20251214 03:02:02.439095 3328 ts_tablet_manager.cc:1929] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:02:02.439182 3328 log.cc:1199] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/2c7821011a7c495fa8506733a7ac1edc
I20251214 03:02:02.439431 3328 ts_tablet_manager.cc:1950] T 2c7821011a7c495fa8506733a7ac1edc P 5e046973a7df4072a4bc289adbf03ce0: Deleting consensus metadata
I20251214 03:02:02.440135 2778 catalog_manager.cc:4985] TS 5e046973a7df4072a4bc289adbf03ce0 (127.1.157.195:41531): tablet 2c7821011a7c495fa8506733a7ac1edc (table test-table [id=046373f8318c4bb3bc317c365091b539]) successfully deleted
Dec 14 03:02:02 dist-test-slave-k7z7 krb5kdc[2743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.157.193@KRBTEST.COM
I20251214 03:02:02.450991 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.433112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.1.157.193:34795)
1214 03:02:02.433271 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.433275 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.441539 (+ 8264us) server_negotiation.cc:408] Connection header received
1214 03:02:02.441680 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.441684 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.441744 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.441829 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:02.442828 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.443572 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.444471 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.445020 (+ 549us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.447272 (+ 2252us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:02.447298 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:02.447311 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:02.447340 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:02.449351 (+ 2011us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:02.449778 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:02.449785 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:02.449787 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:02.449852 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:02.450061 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:02.450064 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:02.450066 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:02.450277 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:02.450483 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.450703 (+ 220us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.450827 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20251214 03:02:02.451421 2990 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36658
I20251214 03:02:02.460623 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:02.453008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.1.157.254:33877)
1214 03:02:02.453126 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:02.453130 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:02.453531 (+ 401us) server_negotiation.cc:408] Connection header received
1214 03:02:02.453633 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:02.453636 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:02.453689 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:02.453768 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:02.454619 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.455340 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:02.456117 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:02.456624 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:02.457494 (+ 870us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:02.457513 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:02.457517 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:02.457543 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:02.459105 (+ 1562us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:02.459518 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:02.459522 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:02.459524 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:02.459568 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:02.459768 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:02.459772 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:02.459774 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:02.459936 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:02.460056 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:02.460302 (+ 246us) server_negotiation.cc:300] Negotiation successful
1214 03:02:02.460462 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20251214 03:02:02.461354 2791 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44898
I20251214 03:02:02.462785 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 2865
I20251214 03:02:02.469218 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 3007
I20251214 03:02:02.475261 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 3149
I20251214 03:02:02.480710 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 2759
2025-12-14T03:02:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3580 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:02:02 dist-test-slave-k7z7 krb5kdc[3366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:02:02 dist-test-slave-k7z7 krb5kdc[3366](info): set up 2 sockets
Dec 14 03:02:02 dist-test-slave-k7z7 krb5kdc[3366](info): commencing operation
krb5kdc: starting...
W20251214 03:02:04.312966 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.811s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:02:04 dist-test-slave-k7z7 krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:02:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:04.444851 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:38199
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:38199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:04.549134 3382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:04.549387 3382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:04.549443 3382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:04.552706 3382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:04.552776 3382 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251214 03:02:04.552794 3382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:04.552814 3382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:04.552832 3382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:04.552847 3382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:04.557317 3382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:38199
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:38199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:04.558398 3382 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:04.559219 3382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:04.564950 3388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:04.564914 3387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:04.564934 3390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:04.565022 3382 server_base.cc:1047] running on GCE node
I20251214 03:02:04.565694 3382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:04.566656 3382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:04.567837 3382 hybrid_clock.cc:648] HybridClock initialized: now 1765681324567806 us; error 51 us; skew 500 ppm
Dec 14 03:02:04 dist-test-slave-k7z7 krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:04.570535 3382 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:02:04.571650 3382 webserver.cc:492] Webserver started at http://127.1.157.254:39947/ using document root <none> and password file <none>
I20251214 03:02:04.572178 3382 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:04.572230 3382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:04.572530 3382 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:04.574354 3382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "0970128eef274e7abb5c45b170a0a822"
format_stamp: "Formatted at 2025-12-14 03:02:04 on dist-test-slave-k7z7"
server_key: "8938fce86a25d35ea244743a62084287"
server_key_iv: "eef6971a27533ae0b4345add96ce5d10"
server_key_version: "encryptionkey@0"
I20251214 03:02:04.574836 3382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "0970128eef274e7abb5c45b170a0a822"
format_stamp: "Formatted at 2025-12-14 03:02:04 on dist-test-slave-k7z7"
server_key: "8938fce86a25d35ea244743a62084287"
server_key_iv: "eef6971a27533ae0b4345add96ce5d10"
server_key_version: "encryptionkey@0"
I20251214 03:02:04.578194 3382 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20251214 03:02:04.580384 3397 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:04.581462 3382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:04.581581 3382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "0970128eef274e7abb5c45b170a0a822"
format_stamp: "Formatted at 2025-12-14 03:02:04 on dist-test-slave-k7z7"
server_key: "8938fce86a25d35ea244743a62084287"
server_key_iv: "eef6971a27533ae0b4345add96ce5d10"
server_key_version: "encryptionkey@0"
I20251214 03:02:04.581672 3382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:04.602030 3382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:04.605410 3382 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:04.605808 3382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:04.616173 3382 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:38199
I20251214 03:02:04.616176 3459 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:38199 every 8 connection(s)
I20251214 03:02:04.617384 3382 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:04.620407 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 3382
I20251214 03:02:04.620520 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:04.620735 1655 external_mini_cluster.cc:1468] Setting key a312d6c2400ff974886e5e10482268ad
I20251214 03:02:04.620725 3460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:04.628196 3460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822: Bootstrap starting.
Dec 14 03:02:04 dist-test-slave-k7z7 krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:04.630640 3460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:04.631505 3460 log.cc:826] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:04.633813 3460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822: No bootstrap required, opened a new log
I20251214 03:02:04.634372 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:04.621958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.1.157.254:38199)
1214 03:02:04.622386 (+ 428us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:04.622399 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:04.622442 (+ 43us) server_negotiation.cc:408] Connection header received
1214 03:02:04.623386 (+ 944us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:04.623415 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:04.624194 (+ 779us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:04.624636 (+ 442us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:04.625501 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:04.626632 (+ 1131us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:04.627290 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:04.627564 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:04.629472 (+ 1908us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:04.629521 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:04.629535 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:04.629578 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:04.631556 (+ 1978us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:04.632008 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:04.632013 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:04.632018 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:04.632092 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:04.632375 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:04.632380 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:04.632382 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:04.632818 (+ 436us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:04.633044 (+ 226us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:04.633357 (+ 313us) server_negotiation.cc:300] Negotiation successful
1214 03:02:04.633625 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20251214 03:02:04.637127 3460 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0970128eef274e7abb5c45b170a0a822" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38199 } }
I20251214 03:02:04.637357 3460 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:04.637403 3460 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0970128eef274e7abb5c45b170a0a822, State: Initialized, Role: FOLLOWER
I20251214 03:02:04.637763 3460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0970128eef274e7abb5c45b170a0a822" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38199 } }
I20251214 03:02:04.637862 3460 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:04.637903 3460 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:04.637965 3460 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:04.638748 3460 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0970128eef274e7abb5c45b170a0a822" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38199 } }
I20251214 03:02:04.639075 3460 leader_election.cc:304] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0970128eef274e7abb5c45b170a0a822; no voters:
I20251214 03:02:04.639369 3460 leader_election.cc:290] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:04.639488 3465 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:04.639678 3465 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [term 1 LEADER]: Becoming Leader. State: Replica: 0970128eef274e7abb5c45b170a0a822, State: Running, Role: LEADER
I20251214 03:02:04.639958 3465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0970128eef274e7abb5c45b170a0a822" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38199 } }
I20251214 03:02:04.640375 3460 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:04.641355 3466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0970128eef274e7abb5c45b170a0a822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0970128eef274e7abb5c45b170a0a822" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38199 } } }
I20251214 03:02:04.641476 3466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:04.641711 3467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0970128eef274e7abb5c45b170a0a822. Latest consensus state: current_term: 1 leader_uuid: "0970128eef274e7abb5c45b170a0a822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0970128eef274e7abb5c45b170a0a822" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38199 } } }
I20251214 03:02:04.641793 3467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:04.641819 3474 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:04.644315 3474 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:04.649343 3474 catalog_manager.cc:1357] Generated new cluster ID: 435c31a93b1447b2986755e249b1f2ea
I20251214 03:02:04.649423 3474 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:04.674413 3474 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:04.675578 3474 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:04.683693 3474 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822: Generated new TSK 0
I20251214 03:02:04.684746 3474 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:04.747962 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:38199
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:04.858278 3488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:04.858516 3488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:04.858589 3488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:04.861992 3488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:04.862068 3488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:04.862131 3488 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251214 03:02:04.862170 3488 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251214 03:02:04.862236 3488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:04.866598 3488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.157.254:38199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:04.867705 3488 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:04.868572 3488 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:04.875357 3494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:04.875622 3488 server_base.cc:1047] running on GCE node
W20251214 03:02:04.875353 3493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:04.875353 3496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:04.876086 3488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:04.876731 3488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:04.877866 3488 hybrid_clock.cc:648] HybridClock initialized: now 1765681324877852 us; error 30 us; skew 500 ppm
Dec 14 03:02:04 dist-test-slave-k7z7 krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:04.880487 3488 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:02:04.881631 3488 webserver.cc:492] Webserver started at http://127.1.157.193:41365/ using document root <none> and password file <none>
I20251214 03:02:04.882174 3488 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:04.882226 3488 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:04.882416 3488 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:04.884292 3488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "9ddec69b8741438385f4a042ca559b2e"
format_stamp: "Formatted at 2025-12-14 03:02:04 on dist-test-slave-k7z7"
server_key: "ed38eabb6c5e8a4b0302b5648da29779"
server_key_iv: "d52e0e9099a296cfbac3c9226fe82881"
server_key_version: "encryptionkey@0"
I20251214 03:02:04.884773 3488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "9ddec69b8741438385f4a042ca559b2e"
format_stamp: "Formatted at 2025-12-14 03:02:04 on dist-test-slave-k7z7"
server_key: "ed38eabb6c5e8a4b0302b5648da29779"
server_key_iv: "d52e0e9099a296cfbac3c9226fe82881"
server_key_version: "encryptionkey@0"
I20251214 03:02:04.888228 3488 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:02:04.890661 3503 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:04.891791 3488 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:04.891909 3488 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "9ddec69b8741438385f4a042ca559b2e"
format_stamp: "Formatted at 2025-12-14 03:02:04 on dist-test-slave-k7z7"
server_key: "ed38eabb6c5e8a4b0302b5648da29779"
server_key_iv: "d52e0e9099a296cfbac3c9226fe82881"
server_key_version: "encryptionkey@0"
I20251214 03:02:04.892009 3488 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:04.904881 3488 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:04.907577 3488 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:04.907748 3488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:04.909264 3488 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:04.909318 3488 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:04.909371 3488 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:04.909399 3488 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 14 03:02:04 dist-test-slave-k7z7 krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:04.919974 3488 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:43107
I20251214 03:02:04.919999 3618 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:43107 every 8 connection(s)
I20251214 03:02:04.920606 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:04.909947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:49219 (local address 127.1.157.254:38199)
1214 03:02:04.910079 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:04.910083 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:04.910903 (+ 820us) server_negotiation.cc:408] Connection header received
1214 03:02:04.911773 (+ 870us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:04.911777 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:04.911841 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:04.911916 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:04.913579 (+ 1663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:04.914159 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:04.914785 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:04.914961 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:04.917002 (+ 2041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:04.917021 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:04.917024 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:04.917047 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:04.918589 (+ 1542us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:04.919104 (+ 515us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:04.919108 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:04.919110 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:04.919172 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:04.919408 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:04.919412 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:04.919413 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:04.919610 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:04.919710 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:04.920349 (+ 639us) server_negotiation.cc:300] Negotiation successful
1214 03:02:04.920483 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251214 03:02:04.921054 3488 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:02:04.921283 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:04.910250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:38199 (local address 127.1.157.193:49219)
1214 03:02:04.910761 (+ 511us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:04.910796 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:04.911575 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:04.912103 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:04.912114 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:04.912652 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:04.913436 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:04.913453 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:04.914275 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:04.914278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:04.914684 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:04.914691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:04.914887 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:04.915494 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:04.915517 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:04.916867 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:04.918709 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:04.918714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:04.918727 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:04.919013 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:04.919261 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:04.919264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:04.919265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:04.919336 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:04.919767 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:04.919775 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:04.920051 (+ 276us) client_negotiation.cc:769] Sending connection context
1214 03:02:04.920274 (+ 223us) client_negotiation.cc:240] Negotiation successful
1214 03:02:04.920529 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":147,"threads_started":1}
I20251214 03:02:04.922607 3619 heartbeater.cc:344] Connected to a master server at 127.1.157.254:38199
I20251214 03:02:04.922780 3619 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:04.923324 3619 heartbeater.cc:507] Master 127.1.157.254:38199 requested a full tablet report, sending...
I20251214 03:02:04.923849 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 3488
I20251214 03:02:04.923964 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:02:04.924222 1655 external_mini_cluster.cc:1468] Setting key c712c0914674a06129289f4ea788bd53
I20251214 03:02:04.924870 3409 ts_manager.cc:194] Registered new tserver with Master: 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107)
I20251214 03:02:04.926082 3409 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:49219
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:02:04.933800 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:04.925486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:48705 (local address 127.1.157.254:38199)
1214 03:02:04.925606 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:04.925611 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:04.926716 (+ 1105us) server_negotiation.cc:408] Connection header received
1214 03:02:04.928258 (+ 1542us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:04.928262 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:04.928314 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:04.928411 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:04.928936 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:04.929450 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:04.929997 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:04.930213 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:04.931022 (+ 809us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:04.931036 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:04.931038 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:04.931059 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:04.932458 (+ 1399us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:04.932813 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:04.932817 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:04.932819 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:04.932859 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:04.933055 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:04.933058 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:04.933060 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:04.933234 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:04.933317 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:04.933565 (+ 248us) server_negotiation.cc:300] Negotiation successful
1214 03:02:04.933660 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251214 03:02:04.933833 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:04.925411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:38199 (local address 127.1.157.193:48705)
1214 03:02:04.926627 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:04.926642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:04.926738 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:04.928449 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:04.928452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:04.928603 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:04.928823 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:04.928829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:04.929544 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:04.929548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:04.929902 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:04.929908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:04.930004 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:04.930613 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:04.930626 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:04.930893 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:04.932563 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:04.932567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:04.932569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:04.932738 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:04.932931 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:04.932934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:04.932937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:04.932986 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:04.933316 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:04.933319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:04.933419 (+ 100us) client_negotiation.cc:769] Sending connection context
1214 03:02:04.933507 (+ 88us) client_negotiation.cc:240] Negotiation successful
1214 03:02:04.933629 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:04.971984 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:38199
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:05.073762 3630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:05.074033 3630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:05.074121 3630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:05.077423 3630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:05.077493 3630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:05.077548 3630 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251214 03:02:05.077569 3630 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251214 03:02:05.077610 3630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:05.081911 3630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.157.254:38199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:05.082990 3630 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:05.083868 3630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:05.091413 3638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:05.091418 3635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:05.091480 3636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:05.091742 3630 server_base.cc:1047] running on GCE node
I20251214 03:02:05.092197 3630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:05.092799 3630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:05.093969 3630 hybrid_clock.cc:648] HybridClock initialized: now 1765681325093954 us; error 33 us; skew 500 ppm
Dec 14 03:02:05 dist-test-slave-k7z7 krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681325, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:05.096679 3630 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:02:05.097718 3630 webserver.cc:492] Webserver started at http://127.1.157.194:42835/ using document root <none> and password file <none>
I20251214 03:02:05.098250 3630 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:05.098301 3630 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:05.098474 3630 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:05.100160 3630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "6464e90933414cd7802684d3e98fd4e4"
format_stamp: "Formatted at 2025-12-14 03:02:05 on dist-test-slave-k7z7"
server_key: "2df5f3c33f96719ba0ba33872ebca185"
server_key_iv: "4c29e50c1b0664ca177d9f63a72ec855"
server_key_version: "encryptionkey@0"
I20251214 03:02:05.100638 3630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "6464e90933414cd7802684d3e98fd4e4"
format_stamp: "Formatted at 2025-12-14 03:02:05 on dist-test-slave-k7z7"
server_key: "2df5f3c33f96719ba0ba33872ebca185"
server_key_iv: "4c29e50c1b0664ca177d9f63a72ec855"
server_key_version: "encryptionkey@0"
I20251214 03:02:05.104255 3630 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251214 03:02:05.106911 3645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:05.108016 3630 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:05.108134 3630 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "6464e90933414cd7802684d3e98fd4e4"
format_stamp: "Formatted at 2025-12-14 03:02:05 on dist-test-slave-k7z7"
server_key: "2df5f3c33f96719ba0ba33872ebca185"
server_key_iv: "4c29e50c1b0664ca177d9f63a72ec855"
server_key_version: "encryptionkey@0"
I20251214 03:02:05.108247 3630 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:05.124835 3630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:05.127583 3630 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:05.127758 3630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:05.129212 3630 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:05.129276 3630 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:05.129331 3630 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:05.129355 3630 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 14 03:02:05 dist-test-slave-k7z7 krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681325, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:05.140033 3630 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:39591
I20251214 03:02:05.140049 3760 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:39591 every 8 connection(s)
I20251214 03:02:05.141219 3630 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:02:05.141395 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.129891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:41659 (local address 127.1.157.254:38199)
1214 03:02:05.130005 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.130009 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.130815 (+ 806us) server_negotiation.cc:408] Connection header received
1214 03:02:05.131669 (+ 854us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.131672 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.131732 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.131804 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:05.133475 (+ 1671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.134253 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.134992 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.135197 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.137689 (+ 2492us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:05.137709 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:05.137711 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:05.137738 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:05.139440 (+ 1702us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.139962 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.139965 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.139967 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.140013 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.140295 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.140298 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.140300 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.140469 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:05.140557 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.141060 (+ 503us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.141237 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251214 03:02:05.141989 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.130234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:38199 (local address 127.1.157.194:41659)
1214 03:02:05.130684 (+ 450us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.130718 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.131478 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:05.131985 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:05.131996 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:05.132561 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:05.133303 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.133320 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.134376 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.134379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:05.134883 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.134892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.135097 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.135748 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:05.135772 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:05.137532 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:05.139556 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:05.139561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:05.139569 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:05.139849 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:05.140094 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:05.140098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:05.140100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:05.140209 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:05.140565 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:05.140571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:05.140800 (+ 229us) client_negotiation.cc:769] Sending connection context
1214 03:02:05.140974 (+ 174us) client_negotiation.cc:240] Negotiation successful
1214 03:02:05.141280 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":145,"threads_started":1}
I20251214 03:02:05.143203 3761 heartbeater.cc:344] Connected to a master server at 127.1.157.254:38199
I20251214 03:02:05.143365 3761 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:05.143844 3761 heartbeater.cc:507] Master 127.1.157.254:38199 requested a full tablet report, sending...
I20251214 03:02:05.144799 3414 ts_manager.cc:194] Registered new tserver with Master: 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591)
I20251214 03:02:05.145447 3414 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:41659
I20251214 03:02:05.147557 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 3630
I20251214 03:02:05.147646 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:02:05.147859 1655 external_mini_cluster.cc:1468] Setting key 07dfd9e915bc5bb18a9019ad04968baf
I20251214 03:02:05.151684 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.145235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:47827 (local address 127.1.157.254:38199)
1214 03:02:05.145411 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.145415 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.145460 (+ 45us) server_negotiation.cc:408] Connection header received
1214 03:02:05.145575 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.145578 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.145622 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.145694 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:05.146334 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.147026 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.147747 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.147937 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.148729 (+ 792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:05.148744 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:05.148747 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:05.148770 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:05.150288 (+ 1518us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.150665 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.150669 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.150670 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.150714 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.150909 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.150912 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.150914 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.151100 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:05.151188 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.151432 (+ 244us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.151546 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20251214 03:02:05.151702 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.145132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:38199 (local address 127.1.157.194:47827)
1214 03:02:05.145389 (+ 257us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.145402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.145499 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:05.145764 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:05.145768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:05.145965 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:05.146214 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.146222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.147194 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.147197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:05.147585 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.147593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.147708 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.148282 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:05.148295 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:05.148622 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:05.150400 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:05.150404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:05.150406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:05.150576 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:05.150790 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:05.150793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:05.150795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:05.150837 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:05.151190 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:05.151201 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:05.151284 (+ 83us) client_negotiation.cc:769] Sending connection context
1214 03:02:05.151370 (+ 86us) client_negotiation.cc:240] Negotiation successful
1214 03:02:05.151511 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:05.193027 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:38199
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:05.295681 3772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:05.295902 3772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:05.295987 3772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:05.299304 3772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:05.299376 3772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:05.299453 3772 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251214 03:02:05.299481 3772 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251214 03:02:05.299538 3772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:05.303797 3772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.157.254:38199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:05.304857 3772 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:05.305687 3772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:05.312582 3778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:05.312685 3780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:05.312582 3777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:05.312983 3772 server_base.cc:1047] running on GCE node
I20251214 03:02:05.313356 3772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:05.313907 3772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:05.315085 3772 hybrid_clock.cc:648] HybridClock initialized: now 1765681325315068 us; error 40 us; skew 500 ppm
Dec 14 03:02:05 dist-test-slave-k7z7 krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681325, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:05.317909 3772 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:02:05.318955 3772 webserver.cc:492] Webserver started at http://127.1.157.195:43277/ using document root <none> and password file <none>
I20251214 03:02:05.319471 3772 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:05.319516 3772 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:05.319674 3772 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:05.321373 3772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "5d3080dfb9f744188845903601db970b"
format_stamp: "Formatted at 2025-12-14 03:02:05 on dist-test-slave-k7z7"
server_key: "3d82d51f8a2090faed8ca01b6d3eae5e"
server_key_iv: "652e8f15851f432abd44925ae0b6758c"
server_key_version: "encryptionkey@0"
I20251214 03:02:05.321811 3772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "5d3080dfb9f744188845903601db970b"
format_stamp: "Formatted at 2025-12-14 03:02:05 on dist-test-slave-k7z7"
server_key: "3d82d51f8a2090faed8ca01b6d3eae5e"
server_key_iv: "652e8f15851f432abd44925ae0b6758c"
server_key_version: "encryptionkey@0"
I20251214 03:02:05.325418 3772 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:02:05.327726 3787 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:05.328902 3772 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:05.329020 3772 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "5d3080dfb9f744188845903601db970b"
format_stamp: "Formatted at 2025-12-14 03:02:05 on dist-test-slave-k7z7"
server_key: "3d82d51f8a2090faed8ca01b6d3eae5e"
server_key_iv: "652e8f15851f432abd44925ae0b6758c"
server_key_version: "encryptionkey@0"
I20251214 03:02:05.329121 3772 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:05.344324 3772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:05.347229 3772 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:05.347411 3772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:05.348888 3772 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:05.348937 3772 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:05.348989 3772 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:05.349018 3772 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 14 03:02:05 dist-test-slave-k7z7 krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681325, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:05.359419 3902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:41381 every 8 connection(s)
I20251214 03:02:05.359398 3772 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:41381
I20251214 03:02:05.360208 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.349460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:37267 (local address 127.1.157.254:38199)
1214 03:02:05.349614 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.349619 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.350298 (+ 679us) server_negotiation.cc:408] Connection header received
1214 03:02:05.351154 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.351158 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.351227 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.351295 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:05.352679 (+ 1384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.353291 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.353913 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.354092 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.356562 (+ 2470us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:05.356583 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:05.356585 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:05.356612 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:05.358297 (+ 1685us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.358786 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.358789 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.358790 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.358829 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.359107 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.359111 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.359112 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.359308 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:05.359401 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.359935 (+ 534us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.360052 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251214 03:02:05.360445 3772 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:02:05.360805 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.349720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:38199 (local address 127.1.157.195:37267)
1214 03:02:05.350153 (+ 433us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.350194 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.350952 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:05.351469 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:05.351480 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:05.351952 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:05.352539 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.352552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.353398 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.353401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:05.353810 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.353817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.354014 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.354645 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:05.354680 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:05.356400 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:05.358393 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:05.358399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:05.358410 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:05.358697 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:05.358900 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:05.358904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:05.358906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:05.359014 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:05.359412 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:05.359419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:05.359701 (+ 282us) client_negotiation.cc:769] Sending connection context
1214 03:02:05.359880 (+ 179us) client_negotiation.cc:240] Negotiation successful
1214 03:02:05.360135 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20251214 03:02:05.362380 3903 heartbeater.cc:344] Connected to a master server at 127.1.157.254:38199
I20251214 03:02:05.362525 3903 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:05.362964 3903 heartbeater.cc:507] Master 127.1.157.254:38199 requested a full tablet report, sending...
I20251214 03:02:05.363866 3409 ts_manager.cc:194] Registered new tserver with Master: 5d3080dfb9f744188845903601db970b (127.1.157.195:41381)
I20251214 03:02:05.364543 3409 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:37267
I20251214 03:02:05.368295 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 3772
I20251214 03:02:05.368410 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:02:05.368659 1655 external_mini_cluster.cc:1468] Setting key 17a8ff35a00abad0c7a68a3147148474
I20251214 03:02:05.370349 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.364165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:38199 (local address 127.1.157.195:56291)
1214 03:02:05.364431 (+ 266us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.364445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.364538 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:05.364774 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:05.364777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:05.364973 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:05.365232 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.365241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.365886 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.365889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:05.366397 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.366408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.366540 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.367057 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:05.367069 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:05.367351 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:05.368929 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:05.368933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:05.368935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:05.369137 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:05.369414 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:05.369445 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:05.369448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:05.369531 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:05.369866 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:05.369870 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:05.369962 (+ 92us) client_negotiation.cc:769] Sending connection context
1214 03:02:05.370062 (+ 100us) client_negotiation.cc:240] Negotiation successful
1214 03:02:05.370178 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20251214 03:02:05.370362 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.364257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:56291 (local address 127.1.157.254:38199)
1214 03:02:05.364417 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.364421 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.364506 (+ 85us) server_negotiation.cc:408] Connection header received
1214 03:02:05.364628 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.364631 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.364678 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.364743 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:05.365340 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.365793 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.366536 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.366724 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.367485 (+ 761us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:05.367499 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:05.367501 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:05.367522 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:05.368833 (+ 1311us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.369272 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.369275 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.369276 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.369335 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.369620 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.369623 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.369625 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.369778 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:05.369872 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.370129 (+ 257us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.370228 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20251214 03:02:05.370800 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 14 03:02:05 dist-test-slave-k7z7 krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 14 03:02:05 dist-test-slave-k7z7 krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:05.393122 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.385812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.1.157.254:38199)
1214 03:02:05.385948 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.385952 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.385964 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:02:05.386002 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.386005 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.386047 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.386115 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:05.386933 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.387422 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.388048 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.388228 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.390350 (+ 2122us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:05.390369 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:05.390372 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:05.390412 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:05.391748 (+ 1336us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.392120 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.392124 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.392125 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.392169 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.392373 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.392376 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.392378 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.392539 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:05.392659 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.392855 (+ 196us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.392966 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251214 03:02:05.396090 3409 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:02:05.398649 3409 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:05.413964 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.409361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:39591 (local address 127.0.0.1:49098)
1214 03:02:05.410043 (+ 682us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.410058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.410139 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:05.410694 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:05.410700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:05.410724 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:05.410984 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.410993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.412505 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.412508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:05.413274 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.413284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.413476 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.413563 (+ 87us) client_negotiation.cc:769] Sending connection context
1214 03:02:05.413685 (+ 122us) client_negotiation.cc:240] Negotiation successful
1214 03:02:05.413784 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"spinlock_wait_cycles":768,"thread_start_us":510,"threads_started":1}
I20251214 03:02:05.414137 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.408567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41381 (local address 127.0.0.1:45186)
1214 03:02:05.409227 (+ 660us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.409268 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.409399 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:05.411042 (+ 1643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:05.411045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:05.411066 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:05.411300 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.411308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.412682 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.412685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:05.413770 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.413780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.413882 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.413901 (+ 19us) client_negotiation.cc:769] Sending connection context
1214 03:02:05.413947 (+ 46us) client_negotiation.cc:240] Negotiation successful
1214 03:02:05.414008 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":159,"threads_started":1}
I20251214 03:02:05.414464 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.409734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.1.157.194:39591)
1214 03:02:05.410307 (+ 573us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.410312 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.410330 (+ 18us) server_negotiation.cc:408] Connection header received
1214 03:02:05.410391 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.410395 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.410544 (+ 149us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.410668 (+ 124us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:05.411129 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.412380 (+ 1251us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.413508 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.414026 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.414149 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.414216 (+ 67us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.414306 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":92,"threads_started":1}
I20251214 03:02:05.415155 3924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.409370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:43107 (local address 127.0.0.1:54586)
1214 03:02:05.410639 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.410753 (+ 114us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.410995 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:05.412222 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:05.412227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:05.412247 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:05.412491 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.412499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.413975 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.413978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:05.414815 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.414825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.414931 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.414948 (+ 17us) client_negotiation.cc:769] Sending connection context
1214 03:02:05.414994 (+ 46us) client_negotiation.cc:240] Negotiation successful
1214 03:02:05.415043 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"mutex_wait_us":422,"thread_start_us":78,"threads_started":1}
I20251214 03:02:05.415468 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.408756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.1.157.195:41381)
1214 03:02:05.409444 (+ 688us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.409454 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.409552 (+ 98us) server_negotiation.cc:408] Connection header received
1214 03:02:05.410056 (+ 504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.410063 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.410804 (+ 741us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.411036 (+ 232us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:05.411423 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.412545 (+ 1122us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.414242 (+ 1697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.414774 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.414873 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.414935 (+ 62us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.415029 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":108,"threads_started":1}
I20251214 03:02:05.416114 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.409598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.1.157.193:43107)
1214 03:02:05.410120 (+ 522us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.410126 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.410909 (+ 783us) server_negotiation.cc:408] Connection header received
1214 03:02:05.411167 (+ 258us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.411173 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.411345 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.411757 (+ 412us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:05.412624 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.413861 (+ 1237us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.415039 (+ 1178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.415747 (+ 708us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.415840 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.415904 (+ 64us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.415984 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":71,"threads_started":1}
I20251214 03:02:05.417954 3695 tablet_service.cc:1505] Processing CreateTablet for tablet f7aff546d1de4500b531aa7be1fe5f95 (DEFAULT_TABLE table=test-table [id=6a12fb4905884867a42b0a570768bc80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:05.418211 3553 tablet_service.cc:1505] Processing CreateTablet for tablet f7aff546d1de4500b531aa7be1fe5f95 (DEFAULT_TABLE table=test-table [id=6a12fb4905884867a42b0a570768bc80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:05.418308 3837 tablet_service.cc:1505] Processing CreateTablet for tablet f7aff546d1de4500b531aa7be1fe5f95 (DEFAULT_TABLE table=test-table [id=6a12fb4905884867a42b0a570768bc80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:05.419029 3553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7aff546d1de4500b531aa7be1fe5f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:05.419034 3695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7aff546d1de4500b531aa7be1fe5f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:05.419034 3837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7aff546d1de4500b531aa7be1fe5f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:05.426697 3927 tablet_bootstrap.cc:492] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: Bootstrap starting.
I20251214 03:02:05.427386 3926 tablet_bootstrap.cc:492] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: Bootstrap starting.
I20251214 03:02:05.427412 3928 tablet_bootstrap.cc:492] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: Bootstrap starting.
I20251214 03:02:05.428907 3927 tablet_bootstrap.cc:654] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:05.429430 3926 tablet_bootstrap.cc:654] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:05.429435 3928 tablet_bootstrap.cc:654] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:05.429989 3927 log.cc:826] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:05.430207 3928 log.cc:826] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:05.430509 3926 log.cc:826] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:05.431946 3927 tablet_bootstrap.cc:492] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: No bootstrap required, opened a new log
I20251214 03:02:05.431946 3928 tablet_bootstrap.cc:492] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: No bootstrap required, opened a new log
I20251214 03:02:05.432126 3927 ts_tablet_manager.cc:1403] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20251214 03:02:05.432126 3928 ts_tablet_manager.cc:1403] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251214 03:02:05.432426 3926 tablet_bootstrap.cc:492] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: No bootstrap required, opened a new log
I20251214 03:02:05.432600 3926 ts_tablet_manager.cc:1403] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251214 03:02:05.435015 3926 raft_consensus.cc:359] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } }
I20251214 03:02:05.435209 3926 raft_consensus.cc:385] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:05.435276 3926 raft_consensus.cc:740] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3080dfb9f744188845903601db970b, State: Initialized, Role: FOLLOWER
I20251214 03:02:05.435252 3927 raft_consensus.cc:359] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } }
I20251214 03:02:05.435472 3927 raft_consensus.cc:385] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:05.435521 3927 raft_consensus.cc:740] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6464e90933414cd7802684d3e98fd4e4, State: Initialized, Role: FOLLOWER
I20251214 03:02:05.435683 3926 consensus_queue.cc:260] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } }
I20251214 03:02:05.435732 3928 raft_consensus.cc:359] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } }
I20251214 03:02:05.435914 3928 raft_consensus.cc:385] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:05.435964 3928 raft_consensus.cc:740] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ddec69b8741438385f4a042ca559b2e, State: Initialized, Role: FOLLOWER
I20251214 03:02:05.435926 3927 consensus_queue.cc:260] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } }
I20251214 03:02:05.436574 3761 heartbeater.cc:499] Master 127.1.157.254:38199 was elected leader, sending a full tablet report...
I20251214 03:02:05.436580 3926 ts_tablet_manager.cc:1434] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251214 03:02:05.436550 3928 consensus_queue.cc:260] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } }
I20251214 03:02:05.436902 3903 heartbeater.cc:499] Master 127.1.157.254:38199 was elected leader, sending a full tablet report...
I20251214 03:02:05.437306 3928 ts_tablet_manager.cc:1434] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251214 03:02:05.437638 3619 heartbeater.cc:499] Master 127.1.157.254:38199 was elected leader, sending a full tablet report...
I20251214 03:02:05.438184 3927 ts_tablet_manager.cc:1434] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20251214 03:02:05.485849 3932 raft_consensus.cc:493] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:05.486037 3932 raft_consensus.cc:515] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } }
I20251214 03:02:05.487268 3932 leader_election.cc:290] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107), 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591)
I20251214 03:02:05.490458 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.487461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:39591 (local address 127.1.157.195:55257)
1214 03:02:05.487733 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.487747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.487835 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:05.488485 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:05.488489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:05.488508 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:05.488742 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.488749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.489477 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.489481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:05.490028 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.490035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.490132 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.490153 (+ 21us) client_negotiation.cc:769] Sending connection context
1214 03:02:05.490200 (+ 47us) client_negotiation.cc:240] Negotiation successful
1214 03:02:05.490265 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
I20251214 03:02:05.490626 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.487430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:43107 (local address 127.1.157.195:52377)
1214 03:02:05.487570 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.487587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.487675 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:05.488429 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:05.488433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:05.488449 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:05.488684 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.488690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.489434 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.489438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:05.490035 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.490042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.490137 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.490163 (+ 26us) client_negotiation.cc:769] Sending connection context
1214 03:02:05.490211 (+ 48us) client_negotiation.cc:240] Negotiation successful
1214 03:02:05.490286 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20251214 03:02:05.490809 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.487986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:52377 (local address 127.1.157.193:43107)
1214 03:02:05.488114 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.488118 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.488135 (+ 17us) server_negotiation.cc:408] Connection header received
1214 03:02:05.488193 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.488196 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.488244 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.488316 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:05.488795 (+ 479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.489327 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.490139 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.490588 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.490626 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.490676 (+ 50us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.490726 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251214 03:02:05.490837 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.487971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:55257 (local address 127.1.157.194:39591)
1214 03:02:05.488161 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.488165 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.488181 (+ 16us) server_negotiation.cc:408] Connection header received
1214 03:02:05.488243 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.488246 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.488299 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.488374 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:05.488856 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.489375 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.490137 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.490596 (+ 459us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.490632 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.490679 (+ 47us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.490731 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251214 03:02:05.491487 3713 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7aff546d1de4500b531aa7be1fe5f95" candidate_uuid: "5d3080dfb9f744188845903601db970b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6464e90933414cd7802684d3e98fd4e4" is_pre_election: true
I20251214 03:02:05.491488 3573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7aff546d1de4500b531aa7be1fe5f95" candidate_uuid: "5d3080dfb9f744188845903601db970b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ddec69b8741438385f4a042ca559b2e" is_pre_election: true
I20251214 03:02:05.491750 3713 raft_consensus.cc:2468] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3080dfb9f744188845903601db970b in term 0.
I20251214 03:02:05.491750 3573 raft_consensus.cc:2468] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3080dfb9f744188845903601db970b in term 0.
I20251214 03:02:05.492183 3791 leader_election.cc:304] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3080dfb9f744188845903601db970b, 9ddec69b8741438385f4a042ca559b2e; no voters:
I20251214 03:02:05.492404 3932 raft_consensus.cc:2804] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:05.492475 3932 raft_consensus.cc:493] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:05.492522 3932 raft_consensus.cc:3060] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:05.493484 3932 raft_consensus.cc:515] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } }
I20251214 03:02:05.493798 3932 leader_election.cc:290] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [CANDIDATE]: Term 1 election: Requested vote from peers 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107), 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591)
I20251214 03:02:05.494087 3573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7aff546d1de4500b531aa7be1fe5f95" candidate_uuid: "5d3080dfb9f744188845903601db970b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ddec69b8741438385f4a042ca559b2e"
I20251214 03:02:05.494113 3713 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7aff546d1de4500b531aa7be1fe5f95" candidate_uuid: "5d3080dfb9f744188845903601db970b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6464e90933414cd7802684d3e98fd4e4"
I20251214 03:02:05.494195 3573 raft_consensus.cc:3060] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:05.494215 3713 raft_consensus.cc:3060] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:05.495059 3713 raft_consensus.cc:2468] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3080dfb9f744188845903601db970b in term 1.
I20251214 03:02:05.495082 3573 raft_consensus.cc:2468] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3080dfb9f744188845903601db970b in term 1.
I20251214 03:02:05.495324 3789 leader_election.cc:304] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3080dfb9f744188845903601db970b, 6464e90933414cd7802684d3e98fd4e4; no voters:
I20251214 03:02:05.495467 3932 raft_consensus.cc:2804] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:05.495683 3932 raft_consensus.cc:697] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 1 LEADER]: Becoming Leader. State: Replica: 5d3080dfb9f744188845903601db970b, State: Running, Role: LEADER
I20251214 03:02:05.495972 3932 consensus_queue.cc:237] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } }
I20251214 03:02:05.498857 3412 catalog_manager.cc:5654] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d3080dfb9f744188845903601db970b (127.1.157.195). New cstate: current_term: 1 leader_uuid: "5d3080dfb9f744188845903601db970b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } health_report { overall_health: UNKNOWN } } }
Dec 14 03:02:05 dist-test-slave-k7z7 krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:05.534385 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.527096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:38199 (local address 127.0.0.1:42950)
1214 03:02:05.527393 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.527405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.527536 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:05.527757 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:05.527760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:05.528203 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:05.528399 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.528406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.529090 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.529094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:05.529509 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:05.529516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.529622 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.530202 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:05.530216 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:05.531575 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:05.533092 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:05.533100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:05.533104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:05.533342 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:05.533540 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:05.533543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:05.533545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:05.533621 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:05.533917 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:05.533922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:05.534025 (+ 103us) client_negotiation.cc:769] Sending connection context
1214 03:02:05.534108 (+ 83us) client_negotiation.cc:240] Negotiation successful
1214 03:02:05.534227 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":135,"threads_started":1}
I20251214 03:02:05.534403 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.527096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.1.157.254:38199)
1214 03:02:05.527265 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:05.527271 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:05.527477 (+ 206us) server_negotiation.cc:408] Connection header received
1214 03:02:05.527621 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:05.527624 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:05.527662 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:05.527725 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:05.528520 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.528993 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:05.529608 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:05.529763 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:05.531684 (+ 1921us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:05.531701 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:05.531703 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:05.531724 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:05.532969 (+ 1245us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.533422 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.533425 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.533426 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.533467 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:05.533691 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:05.533693 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:05.533694 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:05.533838 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:05.533909 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:05.534138 (+ 229us) server_negotiation.cc:300] Negotiation successful
1214 03:02:05.534289 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":70}
I20251214 03:02:05.537992 3412 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:42950:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251214 03:02:05.538695 3412 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:05.543516 3553 tablet_service.cc:1505] Processing CreateTablet for tablet b26c20e37dc5419d8d420f222f245a20 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa336d86dcea4ddf89bf1b2f90a0e57e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251214 03:02:05.543572 3695 tablet_service.cc:1505] Processing CreateTablet for tablet b26c20e37dc5419d8d420f222f245a20 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa336d86dcea4ddf89bf1b2f90a0e57e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251214 03:02:05.543757 3837 tablet_service.cc:1505] Processing CreateTablet for tablet b26c20e37dc5419d8d420f222f245a20 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa336d86dcea4ddf89bf1b2f90a0e57e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251214 03:02:05.544184 3695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b26c20e37dc5419d8d420f222f245a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:05.544193 3837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b26c20e37dc5419d8d420f222f245a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:05.544194 3553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b26c20e37dc5419d8d420f222f245a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:05.546696 3927 tablet_bootstrap.cc:492] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: Bootstrap starting.
I20251214 03:02:05.546736 3926 tablet_bootstrap.cc:492] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: Bootstrap starting.
I20251214 03:02:05.546746 3928 tablet_bootstrap.cc:492] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: Bootstrap starting.
I20251214 03:02:05.547596 3926 tablet_bootstrap.cc:654] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:05.547719 3928 tablet_bootstrap.cc:654] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:05.547717 3927 tablet_bootstrap.cc:654] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:05.548859 3926 tablet_bootstrap.cc:492] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: No bootstrap required, opened a new log
I20251214 03:02:05.548944 3926 ts_tablet_manager.cc:1403] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:05.549131 3927 tablet_bootstrap.cc:492] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: No bootstrap required, opened a new log
I20251214 03:02:05.549249 3927 ts_tablet_manager.cc:1403] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251214 03:02:05.549454 3928 tablet_bootstrap.cc:492] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: No bootstrap required, opened a new log
I20251214 03:02:05.549428 3926 raft_consensus.cc:359] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:05.549537 3928 ts_tablet_manager.cc:1403] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251214 03:02:05.549543 3926 raft_consensus.cc:385] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:05.549578 3926 raft_consensus.cc:740] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3080dfb9f744188845903601db970b, State: Initialized, Role: FOLLOWER
I20251214 03:02:05.549678 3927 raft_consensus.cc:359] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:05.549714 3926 consensus_queue.cc:260] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:05.549785 3927 raft_consensus.cc:385] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:05.549821 3927 raft_consensus.cc:740] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6464e90933414cd7802684d3e98fd4e4, State: Initialized, Role: FOLLOWER
I20251214 03:02:05.549959 3926 ts_tablet_manager.cc:1434] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:05.549929 3927 consensus_queue.cc:260] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:05.549980 3928 raft_consensus.cc:359] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:05.550086 3928 raft_consensus.cc:385] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:05.550122 3928 raft_consensus.cc:740] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ddec69b8741438385f4a042ca559b2e, State: Initialized, Role: FOLLOWER
I20251214 03:02:05.550163 3927 ts_tablet_manager.cc:1434] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 03:02:05.550248 3928 consensus_queue.cc:260] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:05.550668 3928 ts_tablet_manager.cc:1434] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 03:02:05.550164 3933 tablet_replica.cc:442] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } }
I20251214 03:02:05.550747 3934 tablet_replica.cc:442] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } }
I20251214 03:02:05.550799 3933 tablet_replica.cc:445] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:05.550841 3934 tablet_replica.cc:445] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:05.551658 3932 tablet_replica.cc:442] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } }
I20251214 03:02:05.551784 3932 tablet_replica.cc:445] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: This TxnStatusTablet replica's current role is: FOLLOWER
W20251214 03:02:05.611301 3904 tablet.cc:2378] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251214 03:02:05.642505 3762 tablet.cc:2378] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251214 03:02:05.672847 3620 tablet.cc:2378] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 03:02:05.927527 3932 raft_consensus.cc:493] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:05.927670 3932 raft_consensus.cc:515] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:05.928089 3932 leader_election.cc:290] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107), 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591)
I20251214 03:02:05.928483 3573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b26c20e37dc5419d8d420f222f245a20" candidate_uuid: "5d3080dfb9f744188845903601db970b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ddec69b8741438385f4a042ca559b2e" is_pre_election: true
I20251214 03:02:05.928560 3713 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b26c20e37dc5419d8d420f222f245a20" candidate_uuid: "5d3080dfb9f744188845903601db970b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6464e90933414cd7802684d3e98fd4e4" is_pre_election: true
I20251214 03:02:05.928628 3573 raft_consensus.cc:2468] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3080dfb9f744188845903601db970b in term 0.
I20251214 03:02:05.928682 3713 raft_consensus.cc:2468] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3080dfb9f744188845903601db970b in term 0.
I20251214 03:02:05.928916 3791 leader_election.cc:304] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3080dfb9f744188845903601db970b, 9ddec69b8741438385f4a042ca559b2e; no voters:
I20251214 03:02:05.929100 3932 raft_consensus.cc:2804] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:05.929163 3932 raft_consensus.cc:493] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:05.929224 3932 raft_consensus.cc:3060] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:05.930049 3932 raft_consensus.cc:515] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:05.930357 3932 leader_election.cc:290] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [CANDIDATE]: Term 1 election: Requested vote from peers 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107), 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591)
I20251214 03:02:05.930709 3713 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b26c20e37dc5419d8d420f222f245a20" candidate_uuid: "5d3080dfb9f744188845903601db970b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6464e90933414cd7802684d3e98fd4e4"
I20251214 03:02:05.930709 3573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b26c20e37dc5419d8d420f222f245a20" candidate_uuid: "5d3080dfb9f744188845903601db970b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ddec69b8741438385f4a042ca559b2e"
I20251214 03:02:05.930827 3713 raft_consensus.cc:3060] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:05.930845 3573 raft_consensus.cc:3060] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:05.931854 3573 raft_consensus.cc:2468] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3080dfb9f744188845903601db970b in term 1.
I20251214 03:02:05.931854 3713 raft_consensus.cc:2468] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3080dfb9f744188845903601db970b in term 1.
I20251214 03:02:05.932148 3789 leader_election.cc:304] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3080dfb9f744188845903601db970b, 6464e90933414cd7802684d3e98fd4e4; no voters:
I20251214 03:02:05.932318 3932 raft_consensus.cc:2804] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:05.932399 3932 raft_consensus.cc:697] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [term 1 LEADER]: Becoming Leader. State: Replica: 5d3080dfb9f744188845903601db970b, State: Running, Role: LEADER
I20251214 03:02:05.932531 3932 consensus_queue.cc:237] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:05.933251 3936 tablet_replica.cc:442] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: TxnStatusTablet state changed. Reason: New leader 5d3080dfb9f744188845903601db970b. Latest consensus state: current_term: 1 leader_uuid: "5d3080dfb9f744188845903601db970b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } }
I20251214 03:02:05.933343 3936 tablet_replica.cc:445] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: This TxnStatusTablet replica's current role is: LEADER
I20251214 03:02:05.933548 3948 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251214 03:02:05.933741 3948 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251214 03:02:05.934342 3414 catalog_manager.cc:5654] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d3080dfb9f744188845903601db970b (127.1.157.195). New cstate: current_term: 1 leader_uuid: "5d3080dfb9f744188845903601db970b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } health_report { overall_health: HEALTHY } } }
I20251214 03:02:05.964735 3936 consensus_queue.cc:1048] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:05.970083 3952 consensus_queue.cc:1048] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:06.003307 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.999633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41381 (local address 127.0.0.1:45194)
1214 03:02:05.999810 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:05.999828 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:05.999944 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:06.000284 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:06.000286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:06.000293 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:02:06.000484 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.000489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.001558 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.001561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:06.001982 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.001989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.002074 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.002115 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:02:06.003035 (+ 920us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:02:06.003052 (+ 17us) client_negotiation.cc:769] Sending connection context
1214 03:02:06.003111 (+ 59us) client_negotiation.cc:240] Negotiation successful
1214 03:02:06.003181 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20251214 03:02:06.003360 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:05.999760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45194 (local address 127.1.157.195:41381)
1214 03:02:06.000009 (+ 249us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:06.000014 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:06.000030 (+ 16us) server_negotiation.cc:408] Connection header received
1214 03:02:06.000109 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:06.000112 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:06.000178 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:06.000285 (+ 107us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:06.000578 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.001424 (+ 846us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.002073 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.002270 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.002328 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:06.002940 (+ 612us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:06.003047 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:06.003133 (+ 86us) server_negotiation.cc:300] Negotiation successful
1214 03:02:06.003190 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
I20251214 03:02:06.007988 3573 raft_consensus.cc:1275] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Refusing update from remote peer 5d3080dfb9f744188845903601db970b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:06.007995 3713 raft_consensus.cc:1275] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Refusing update from remote peer 5d3080dfb9f744188845903601db970b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:06.008451 3952 consensus_queue.cc:1048] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:06.008611 3936 consensus_queue.cc:1048] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:06.011499 3933 tablet_replica.cc:442] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: TxnStatusTablet state changed. Reason: New leader 5d3080dfb9f744188845903601db970b. Latest consensus state: current_term: 1 leader_uuid: "5d3080dfb9f744188845903601db970b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } }
I20251214 03:02:06.011615 3933 tablet_replica.cc:445] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:06.012491 3956 tablet_replica.cc:442] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: TxnStatusTablet state changed. Reason: New leader 5d3080dfb9f744188845903601db970b. Latest consensus state: current_term: 1 leader_uuid: "5d3080dfb9f744188845903601db970b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } }
I20251214 03:02:06.012609 3956 tablet_replica.cc:445] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:06.014689 3952 tablet_replica.cc:442] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d3080dfb9f744188845903601db970b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } }
I20251214 03:02:06.014801 3952 tablet_replica.cc:445] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: This TxnStatusTablet replica's current role is: LEADER
I20251214 03:02:06.014904 3952 tablet_replica.cc:442] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d3080dfb9f744188845903601db970b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } }
I20251214 03:02:06.014959 3952 tablet_replica.cc:445] T b26c20e37dc5419d8d420f222f245a20 P 5d3080dfb9f744188845903601db970b: This TxnStatusTablet replica's current role is: LEADER
I20251214 03:02:06.015143 3953 mvcc.cc:204] Tried to move back new op lower bound from 7232230711323545600 to 7232230711021121536. Current Snapshot: MvccSnapshot[applied={T|T < 7232230711323545600}]
I20251214 03:02:06.015301 3933 tablet_replica.cc:442] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d3080dfb9f744188845903601db970b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } }
I20251214 03:02:06.015395 3933 tablet_replica.cc:445] T b26c20e37dc5419d8d420f222f245a20 P 6464e90933414cd7802684d3e98fd4e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:06.015862 3956 tablet_replica.cc:442] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d3080dfb9f744188845903601db970b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } } }
I20251214 03:02:06.015959 3956 tablet_replica.cc:445] T b26c20e37dc5419d8d420f222f245a20 P 9ddec69b8741438385f4a042ca559b2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:02:06.017637 3955 mvcc.cc:204] Tried to move back new op lower bound from 7232230711323545600 to 7232230711021121536. Current Snapshot: MvccSnapshot[applied={T|T < 7232230711323545600}]
I20251214 03:02:06.024544 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.021525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.1.157.195:41381)
1214 03:02:06.021640 (+ 115us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:06.021644 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:06.021812 (+ 168us) server_negotiation.cc:408] Connection header received
1214 03:02:06.021926 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:06.021929 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:06.021982 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:06.022060 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:06.022396 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.023180 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.023888 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.024082 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.024128 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:06.024206 (+ 78us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:06.024279 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:06.024362 (+ 83us) server_negotiation.cc:300] Negotiation successful
1214 03:02:06.024406 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251214 03:02:06.034243 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.031404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:55463 (local address 127.1.157.195:41381)
1214 03:02:06.031514 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:06.031518 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:06.031720 (+ 202us) server_negotiation.cc:408] Connection header received
1214 03:02:06.031830 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:06.031833 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:06.031866 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:06.031920 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:06.032539 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.032999 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.033655 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.033813 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.033848 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:06.033918 (+ 70us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:06.033975 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:06.034065 (+ 90us) server_negotiation.cc:300] Negotiation successful
1214 03:02:06.034099 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251214 03:02:06.034260 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.031345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41381 (local address 127.1.157.195:55463)
1214 03:02:06.031649 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:06.031660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:06.031750 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:06.031933 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:06.031935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:06.031942 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:02:06.032438 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.032445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.033087 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.033091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:06.033545 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.033552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.033646 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.033688 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:02:06.033987 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:02:06.033991 (+ 4us) client_negotiation.cc:769] Sending connection context
1214 03:02:06.034047 (+ 56us) client_negotiation.cc:240] Negotiation successful
1214 03:02:06.034115 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":84,"threads_started":1}
I20251214 03:02:06.074024 3412 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42948:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:06.074211 3412 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42948:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:06.076479 3412 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822: Sending DeleteTablet for 3 replicas of tablet f7aff546d1de4500b531aa7be1fe5f95
I20251214 03:02:06.077208 3553 tablet_service.cc:1552] Processing DeleteTablet for tablet f7aff546d1de4500b531aa7be1fe5f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:06 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:54586
I20251214 03:02:06.077262 3695 tablet_service.cc:1552] Processing DeleteTablet for tablet f7aff546d1de4500b531aa7be1fe5f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:06 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:49098
I20251214 03:02:06.077278 3836 tablet_service.cc:1552] Processing DeleteTablet for tablet f7aff546d1de4500b531aa7be1fe5f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:06 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:45186
I20251214 03:02:06.077682 3972 tablet_replica.cc:333] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: stopping tablet replica
I20251214 03:02:06.077842 3971 tablet_replica.cc:333] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: stopping tablet replica
I20251214 03:02:06.077930 3972 raft_consensus.cc:2243] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:02:06.078148 3971 raft_consensus.cc:2243] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:06.078208 3412 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 03:02:06.078332 3972 raft_consensus.cc:2272] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:06.078555 3971 raft_consensus.cc:2272] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 03:02:06.078627 3412 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:06.079494 3972 ts_tablet_manager.cc:1916] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:06.079859 3973 tablet_replica.cc:333] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: stopping tablet replica
I20251214 03:02:06.080076 3973 raft_consensus.cc:2243] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:06.080272 3973 raft_consensus.cc:2272] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:06.080269 3971 ts_tablet_manager.cc:1916] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:06.083282 3553 tablet_service.cc:1505] Processing CreateTablet for tablet e7977eb2e04e420b9ebbc50fbcad38f7 (DEFAULT_TABLE table=test-table [id=ad920fdf69ae49d58ba96f0ead69e35f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:06.083544 3553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7977eb2e04e420b9ebbc50fbcad38f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:06.083729 3695 tablet_service.cc:1505] Processing CreateTablet for tablet e7977eb2e04e420b9ebbc50fbcad38f7 (DEFAULT_TABLE table=test-table [id=ad920fdf69ae49d58ba96f0ead69e35f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:06.083963 3695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7977eb2e04e420b9ebbc50fbcad38f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:06.084326 3972 ts_tablet_manager.cc:1929] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:02:06.084400 3972 log.cc:1199] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/f7aff546d1de4500b531aa7be1fe5f95
I20251214 03:02:06.084681 3972 ts_tablet_manager.cc:1950] T f7aff546d1de4500b531aa7be1fe5f95 P 5d3080dfb9f744188845903601db970b: Deleting consensus metadata
I20251214 03:02:06.086004 3401 catalog_manager.cc:4985] TS 5d3080dfb9f744188845903601db970b (127.1.157.195:41381): tablet f7aff546d1de4500b531aa7be1fe5f95 (table test-table [id=6a12fb4905884867a42b0a570768bc80]) successfully deleted
I20251214 03:02:06.086591 3836 tablet_service.cc:1505] Processing CreateTablet for tablet e7977eb2e04e420b9ebbc50fbcad38f7 (DEFAULT_TABLE table=test-table [id=ad920fdf69ae49d58ba96f0ead69e35f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:06.086663 3975 tablet_bootstrap.cc:492] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4: Bootstrap starting.
I20251214 03:02:06.086820 3836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7977eb2e04e420b9ebbc50fbcad38f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:06.087232 3971 ts_tablet_manager.cc:1929] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:02:06.087370 3971 log.cc:1199] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/wal/wals/f7aff546d1de4500b531aa7be1fe5f95
I20251214 03:02:06.087604 3975 tablet_bootstrap.cc:654] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:06.087783 3971 ts_tablet_manager.cc:1950] T f7aff546d1de4500b531aa7be1fe5f95 P 9ddec69b8741438385f4a042ca559b2e: Deleting consensus metadata
I20251214 03:02:06.088711 3975 tablet_bootstrap.cc:492] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4: No bootstrap required, opened a new log
I20251214 03:02:06.088804 3975 ts_tablet_manager.cc:1403] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:06.089126 3401 catalog_manager.cc:4985] TS 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107): tablet f7aff546d1de4500b531aa7be1fe5f95 (table test-table [id=6a12fb4905884867a42b0a570768bc80]) successfully deleted
I20251214 03:02:06.089313 3975 raft_consensus.cc:359] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:06.089426 3975 raft_consensus.cc:385] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:06.089463 3975 raft_consensus.cc:740] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6464e90933414cd7802684d3e98fd4e4, State: Initialized, Role: FOLLOWER
I20251214 03:02:06.089586 3975 consensus_queue.cc:260] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:06.089725 3974 tablet_bootstrap.cc:492] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e: Bootstrap starting.
I20251214 03:02:06.089830 3975 ts_tablet_manager.cc:1434] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:06.090070 3973 ts_tablet_manager.cc:1916] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:06.091106 3974 tablet_bootstrap.cc:654] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:06.092337 3974 tablet_bootstrap.cc:492] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e: No bootstrap required, opened a new log
I20251214 03:02:06.092425 3974 ts_tablet_manager.cc:1403] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251214 03:02:06.092664 3973 ts_tablet_manager.cc:1929] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:02:06.092739 3973 log.cc:1199] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/f7aff546d1de4500b531aa7be1fe5f95
I20251214 03:02:06.092887 3974 raft_consensus.cc:359] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:06.092989 3974 raft_consensus.cc:385] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:06.093020 3974 raft_consensus.cc:740] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ddec69b8741438385f4a042ca559b2e, State: Initialized, Role: FOLLOWER
I20251214 03:02:06.093025 3973 ts_tablet_manager.cc:1950] T f7aff546d1de4500b531aa7be1fe5f95 P 6464e90933414cd7802684d3e98fd4e4: Deleting consensus metadata
I20251214 03:02:06.093071 3977 tablet_bootstrap.cc:492] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b: Bootstrap starting.
I20251214 03:02:06.093150 3974 consensus_queue.cc:260] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:06.093452 3974 ts_tablet_manager.cc:1434] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:06.094120 3399 catalog_manager.cc:4985] TS 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591): tablet f7aff546d1de4500b531aa7be1fe5f95 (table test-table [id=6a12fb4905884867a42b0a570768bc80]) successfully deleted
I20251214 03:02:06.094303 3977 tablet_bootstrap.cc:654] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:06.095808 3977 tablet_bootstrap.cc:492] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b: No bootstrap required, opened a new log
I20251214 03:02:06.095882 3977 ts_tablet_manager.cc:1403] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251214 03:02:06.096498 3977 raft_consensus.cc:359] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:06.096589 3977 raft_consensus.cc:385] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:06.096611 3977 raft_consensus.cc:740] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3080dfb9f744188845903601db970b, State: Initialized, Role: FOLLOWER
I20251214 03:02:06.096714 3977 consensus_queue.cc:260] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:06.096956 3977 ts_tablet_manager.cc:1434] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 03:02:06.101356 3933 raft_consensus.cc:493] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:06.101480 3933 raft_consensus.cc:515] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:06.102515 3933 leader_election.cc:290] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107), 5d3080dfb9f744188845903601db970b (127.1.157.195:41381)
I20251214 03:02:06.106084 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.102674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:43107 (local address 127.1.157.194:37619)
1214 03:02:06.102957 (+ 283us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:06.102972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:06.103062 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:06.103932 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:06.103935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:06.103955 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:06.104216 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.104224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.105024 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.105027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:06.105675 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.105683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.105800 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.105825 (+ 25us) client_negotiation.cc:769] Sending connection context
1214 03:02:06.105875 (+ 50us) client_negotiation.cc:240] Negotiation successful
1214 03:02:06.105932 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":124,"threads_started":1}
I20251214 03:02:06.106374 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.103050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41381 (local address 127.1.157.194:34629)
1214 03:02:06.103283 (+ 233us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:06.103295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:06.103374 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:06.103721 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:06.103725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:06.103749 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:06.104052 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.104060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.105062 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.105066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:06.106003 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.106015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.106116 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.106132 (+ 16us) client_negotiation.cc:769] Sending connection context
1214 03:02:06.106172 (+ 40us) client_negotiation.cc:240] Negotiation successful
1214 03:02:06.106237 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20251214 03:02:06.106585 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.103407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:37619 (local address 127.1.157.193:43107)
1214 03:02:06.103703 (+ 296us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:06.103707 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:06.103720 (+ 13us) server_negotiation.cc:408] Connection header received
1214 03:02:06.103766 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:06.103769 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:06.103812 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:06.104309 (+ 497us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:06.104393 (+ 84us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.104919 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.105789 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.106262 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.106311 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:06.106384 (+ 73us) server_negotiation.cc:300] Negotiation successful
1214 03:02:06.106449 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":118,"threads_started":1}
I20251214 03:02:06.106912 3573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7977eb2e04e420b9ebbc50fbcad38f7" candidate_uuid: "6464e90933414cd7802684d3e98fd4e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ddec69b8741438385f4a042ca559b2e" is_pre_election: true
I20251214 03:02:06.107040 3573 raft_consensus.cc:2468] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6464e90933414cd7802684d3e98fd4e4 in term 0.
I20251214 03:02:06.107344 3649 leader_election.cc:304] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6464e90933414cd7802684d3e98fd4e4, 9ddec69b8741438385f4a042ca559b2e; no voters:
I20251214 03:02:06.107556 3933 raft_consensus.cc:2804] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:06.107605 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.103393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:34629 (local address 127.1.157.195:41381)
1214 03:02:06.103508 (+ 115us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:06.103511 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:06.103521 (+ 10us) server_negotiation.cc:408] Connection header received
1214 03:02:06.103559 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:06.103562 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:06.103608 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:06.103683 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:06.104211 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.104934 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.106910 (+ 1976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.107374 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.107407 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:06.107451 (+ 44us) server_negotiation.cc:300] Negotiation successful
1214 03:02:06.107499 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251214 03:02:06.107613 3933 raft_consensus.cc:493] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:06.107656 3933 raft_consensus.cc:3060] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:06.108078 3857 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7977eb2e04e420b9ebbc50fbcad38f7" candidate_uuid: "6464e90933414cd7802684d3e98fd4e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3080dfb9f744188845903601db970b" is_pre_election: true
I20251214 03:02:06.108269 3857 raft_consensus.cc:2468] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6464e90933414cd7802684d3e98fd4e4 in term 0.
I20251214 03:02:06.108796 3933 raft_consensus.cc:515] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:06.109123 3933 leader_election.cc:290] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [CANDIDATE]: Term 1 election: Requested vote from peers 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107), 5d3080dfb9f744188845903601db970b (127.1.157.195:41381)
I20251214 03:02:06.109544 3857 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7977eb2e04e420b9ebbc50fbcad38f7" candidate_uuid: "6464e90933414cd7802684d3e98fd4e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3080dfb9f744188845903601db970b"
I20251214 03:02:06.109544 3573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7977eb2e04e420b9ebbc50fbcad38f7" candidate_uuid: "6464e90933414cd7802684d3e98fd4e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ddec69b8741438385f4a042ca559b2e"
I20251214 03:02:06.109653 3573 raft_consensus.cc:3060] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:06.109653 3857 raft_consensus.cc:3060] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:06.110651 3857 raft_consensus.cc:2468] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6464e90933414cd7802684d3e98fd4e4 in term 1.
I20251214 03:02:06.110652 3573 raft_consensus.cc:2468] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6464e90933414cd7802684d3e98fd4e4 in term 1.
I20251214 03:02:06.110929 3649 leader_election.cc:304] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6464e90933414cd7802684d3e98fd4e4, 9ddec69b8741438385f4a042ca559b2e; no voters:
I20251214 03:02:06.111150 3933 raft_consensus.cc:2804] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:06.111364 3933 raft_consensus.cc:697] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 1 LEADER]: Becoming Leader. State: Replica: 6464e90933414cd7802684d3e98fd4e4, State: Running, Role: LEADER
I20251214 03:02:06.111626 3933 consensus_queue.cc:237] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:06.114058 3409 catalog_manager.cc:5654] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194). New cstate: current_term: 1 leader_uuid: "6464e90933414cd7802684d3e98fd4e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } health_report { overall_health: UNKNOWN } } }
I20251214 03:02:06.132835 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.129793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.1.157.194:39591)
1214 03:02:06.130094 (+ 301us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:06.130097 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:06.130106 (+ 9us) server_negotiation.cc:408] Connection header received
1214 03:02:06.130165 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:06.130168 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:06.130226 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:06.130317 (+ 91us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:06.130662 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.131181 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.131796 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.131974 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.132026 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:06.132449 (+ 423us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:06.132552 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:06.132629 (+ 77us) server_negotiation.cc:300] Negotiation successful
1214 03:02:06.132688 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":134,"threads_started":1}
I20251214 03:02:06.142941 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.139510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:38331 (local address 127.1.157.195:41381)
1214 03:02:06.139637 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:06.139641 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:06.139781 (+ 140us) server_negotiation.cc:408] Connection header received
1214 03:02:06.139985 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:06.139988 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:06.140044 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:06.140122 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:06.140587 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.141313 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.142225 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.142461 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.142511 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:06.142590 (+ 79us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:06.142660 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:06.142776 (+ 116us) server_negotiation.cc:300] Negotiation successful
1214 03:02:06.142821 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251214 03:02:06.142980 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.139398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41381 (local address 127.1.157.194:38331)
1214 03:02:06.139698 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:06.139714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:06.139841 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:06.140138 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:06.140142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:06.140151 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:02:06.140470 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.140478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.141418 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.141422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:06.142110 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.142121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.142240 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.142281 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:02:06.142681 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:02:06.142687 (+ 6us) client_negotiation.cc:769] Sending connection context
1214 03:02:06.142757 (+ 70us) client_negotiation.cc:240] Negotiation successful
1214 03:02:06.142830 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":114,"threads_started":1}
I20251214 03:02:06.149825 3857 raft_consensus.cc:1275] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Refusing update from remote peer 6464e90933414cd7802684d3e98fd4e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:06.150228 3573 raft_consensus.cc:1275] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Refusing update from remote peer 6464e90933414cd7802684d3e98fd4e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:06.150429 3983 consensus_queue.cc:1048] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:06.150969 3983 consensus_queue.cc:1048] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:06.157986 3951 mvcc.cc:204] Tried to move back new op lower bound from 7232230711904399360 to 7232230711756038144. Current Snapshot: MvccSnapshot[applied={T|T < 7232230711904399360}]
I20251214 03:02:06.172037 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.169109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:47135 (local address 127.1.157.194:39591)
1214 03:02:06.169274 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:06.169278 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:06.169294 (+ 16us) server_negotiation.cc:408] Connection header received
1214 03:02:06.169414 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:06.169417 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:06.169490 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:06.169560 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:06.169963 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.170427 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.171383 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.171565 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.171608 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:06.171680 (+ 72us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:06.171751 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:06.171848 (+ 97us) server_negotiation.cc:300] Negotiation successful
1214 03:02:06.171907 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20251214 03:02:06.172040 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:06.169054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:39591 (local address 127.1.157.195:47135)
1214 03:02:06.169188 (+ 134us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:06.169233 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:06.169310 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:06.169589 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:06.169593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:06.169604 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:02:06.169856 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.169863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.170521 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:06.170524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:06.171170 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:06.171179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:06.171288 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:06.171319 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:02:06.171756 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:02:06.171762 (+ 6us) client_negotiation.cc:769] Sending connection context
1214 03:02:06.171831 (+ 69us) client_negotiation.cc:240] Negotiation successful
1214 03:02:06.171907 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
W20251214 03:02:06.172749 3953 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251214 03:02:07.728390 3409 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42948:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:07.728544 3409 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42948:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:07.730744 3409 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822: Sending DeleteTablet for 3 replicas of tablet e7977eb2e04e420b9ebbc50fbcad38f7
I20251214 03:02:07.731274 3553 tablet_service.cc:1552] Processing DeleteTablet for tablet e7977eb2e04e420b9ebbc50fbcad38f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:07 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:54586
I20251214 03:02:07.731312 3836 tablet_service.cc:1552] Processing DeleteTablet for tablet e7977eb2e04e420b9ebbc50fbcad38f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:07 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:45186
I20251214 03:02:07.731329 3695 tablet_service.cc:1552] Processing DeleteTablet for tablet e7977eb2e04e420b9ebbc50fbcad38f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:07 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:49098
I20251214 03:02:07.731622 4015 tablet_replica.cc:333] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e: stopping tablet replica
I20251214 03:02:07.731747 4017 tablet_replica.cc:333] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b: stopping tablet replica
I20251214 03:02:07.731781 4016 tablet_replica.cc:333] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4: stopping tablet replica
I20251214 03:02:07.731902 4017 raft_consensus.cc:2243] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:07.731914 4016 raft_consensus.cc:2243] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:02:07.732059 4017 raft_consensus.cc:2272] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:07.732195 4016 raft_consensus.cc:2272] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:07.732378 4015 raft_consensus.cc:2243] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:07.732513 4015 raft_consensus.cc:2272] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:07.733001 4016 ts_tablet_manager.cc:1916] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:07.733549 4015 ts_tablet_manager.cc:1916] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:07.734732 4017 ts_tablet_manager.cc:1916] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:07.735590 4016 ts_tablet_manager.cc:1929] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251214 03:02:07.735714 4016 log.cc:1199] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/e7977eb2e04e420b9ebbc50fbcad38f7
I20251214 03:02:07.735944 4015 ts_tablet_manager.cc:1929] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251214 03:02:07.736050 4015 log.cc:1199] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/wal/wals/e7977eb2e04e420b9ebbc50fbcad38f7
I20251214 03:02:07.736109 4016 ts_tablet_manager.cc:1950] T e7977eb2e04e420b9ebbc50fbcad38f7 P 6464e90933414cd7802684d3e98fd4e4: Deleting consensus metadata
I20251214 03:02:07.736383 4015 ts_tablet_manager.cc:1950] T e7977eb2e04e420b9ebbc50fbcad38f7 P 9ddec69b8741438385f4a042ca559b2e: Deleting consensus metadata
I20251214 03:02:07.736835 3399 catalog_manager.cc:4985] TS 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591): tablet e7977eb2e04e420b9ebbc50fbcad38f7 (table test-table [id=ad920fdf69ae49d58ba96f0ead69e35f]) successfully deleted
I20251214 03:02:07.737017 3401 catalog_manager.cc:4985] TS 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107): tablet e7977eb2e04e420b9ebbc50fbcad38f7 (table test-table [id=ad920fdf69ae49d58ba96f0ead69e35f]) successfully deleted
I20251214 03:02:07.737128 4017 ts_tablet_manager.cc:1929] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251214 03:02:07.737207 4017 log.cc:1199] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/e7977eb2e04e420b9ebbc50fbcad38f7
I20251214 03:02:07.737421 4017 ts_tablet_manager.cc:1950] T e7977eb2e04e420b9ebbc50fbcad38f7 P 5d3080dfb9f744188845903601db970b: Deleting consensus metadata
I20251214 03:02:07.737957 3401 catalog_manager.cc:4985] TS 5d3080dfb9f744188845903601db970b (127.1.157.195:41381): tablet e7977eb2e04e420b9ebbc50fbcad38f7 (table test-table [id=ad920fdf69ae49d58ba96f0ead69e35f]) successfully deleted
Dec 14 03:02:07 dist-test-slave-k7z7 krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681327, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 14 03:02:07 dist-test-slave-k7z7 krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681327, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:07.758378 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:07.747260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.1.157.254:38199)
1214 03:02:07.747497 (+ 237us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:07.747501 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:07.747514 (+ 13us) server_negotiation.cc:408] Connection header received
1214 03:02:07.747550 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:07.747553 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:07.747595 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:07.747670 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:07.748365 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:07.748898 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:07.749577 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:07.749729 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:07.755989 (+ 6260us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:07.756010 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:07.756012 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:07.756043 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:07.757303 (+ 1260us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:07.757615 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:07.757617 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:07.757619 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:07.757655 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:07.757787 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:07.757789 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:07.757790 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:07.757929 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:07.758017 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:07.758146 (+ 129us) server_negotiation.cc:300] Negotiation successful
1214 03:02:07.758249 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20251214 03:02:07.760564 3409 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:02:07.760973 3409 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:07.765376 3553 tablet_service.cc:1505] Processing CreateTablet for tablet 922cd2aee60248898b1dab8ec200a56e (DEFAULT_TABLE table=test-table [id=eda7fad749ee437cb9d427e9965e26af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:07.765417 3695 tablet_service.cc:1505] Processing CreateTablet for tablet 922cd2aee60248898b1dab8ec200a56e (DEFAULT_TABLE table=test-table [id=eda7fad749ee437cb9d427e9965e26af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:07.765614 3836 tablet_service.cc:1505] Processing CreateTablet for tablet 922cd2aee60248898b1dab8ec200a56e (DEFAULT_TABLE table=test-table [id=eda7fad749ee437cb9d427e9965e26af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:07.765679 3553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 922cd2aee60248898b1dab8ec200a56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:07.765692 3695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 922cd2aee60248898b1dab8ec200a56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:07.765836 3836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 922cd2aee60248898b1dab8ec200a56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:07.767964 4026 tablet_bootstrap.cc:492] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e: Bootstrap starting.
I20251214 03:02:07.767985 4027 tablet_bootstrap.cc:492] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4: Bootstrap starting.
I20251214 03:02:07.768517 4028 tablet_bootstrap.cc:492] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b: Bootstrap starting.
I20251214 03:02:07.768841 4026 tablet_bootstrap.cc:654] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:07.769296 4028 tablet_bootstrap.cc:654] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:07.769984 4026 tablet_bootstrap.cc:492] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e: No bootstrap required, opened a new log
I20251214 03:02:07.770007 4027 tablet_bootstrap.cc:654] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:07.770076 4026 ts_tablet_manager.cc:1403] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251214 03:02:07.770337 4028 tablet_bootstrap.cc:492] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b: No bootstrap required, opened a new log
I20251214 03:02:07.770465 4028 ts_tablet_manager.cc:1403] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:07.770632 4026 raft_consensus.cc:359] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:07.770787 4026 raft_consensus.cc:385] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:07.770839 4026 raft_consensus.cc:740] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ddec69b8741438385f4a042ca559b2e, State: Initialized, Role: FOLLOWER
I20251214 03:02:07.771040 4027 tablet_bootstrap.cc:492] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4: No bootstrap required, opened a new log
I20251214 03:02:07.770994 4026 consensus_queue.cc:260] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:07.771126 4027 ts_tablet_manager.cc:1403] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251214 03:02:07.771164 4028 raft_consensus.cc:359] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:07.771262 4028 raft_consensus.cc:385] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:07.771293 4028 raft_consensus.cc:740] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3080dfb9f744188845903601db970b, State: Initialized, Role: FOLLOWER
I20251214 03:02:07.771366 4026 ts_tablet_manager.cc:1434] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:07.771440 4028 consensus_queue.cc:260] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:07.771510 4027 raft_consensus.cc:359] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:07.771605 4027 raft_consensus.cc:385] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:07.771637 4027 raft_consensus.cc:740] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6464e90933414cd7802684d3e98fd4e4, State: Initialized, Role: FOLLOWER
I20251214 03:02:07.771672 4028 ts_tablet_manager.cc:1434] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:07.771747 4027 consensus_queue.cc:260] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:07.771955 4027 ts_tablet_manager.cc:1434] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:07.943655 4032 raft_consensus.cc:493] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:07.943779 4032 raft_consensus.cc:515] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:07.944842 4032 leader_election.cc:290] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591), 5d3080dfb9f744188845903601db970b (127.1.157.195:41381)
I20251214 03:02:07.948735 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:07.945093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41381 (local address 127.1.157.193:40737)
1214 03:02:07.945494 (+ 401us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:07.945509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:07.945656 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:07.945924 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:07.945927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:07.945952 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:07.946291 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:07.946301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:07.947335 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:07.947339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:07.948301 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:07.948312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:07.948429 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:07.948451 (+ 22us) client_negotiation.cc:769] Sending connection context
1214 03:02:07.948501 (+ 50us) client_negotiation.cc:240] Negotiation successful
1214 03:02:07.948563 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":28,"spinlock_wait_cycles":30848,"thread_start_us":136,"threads_started":1}
I20251214 03:02:07.948738 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:07.945088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:39591 (local address 127.1.157.193:37733)
1214 03:02:07.945494 (+ 406us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:07.945509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:07.945648 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:07.945924 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:07.945927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:07.945955 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:07.946296 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:07.946301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:07.947335 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:07.947339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:07.948301 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:07.948312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:07.948438 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:07.948451 (+ 13us) client_negotiation.cc:769] Sending connection context
1214 03:02:07.948501 (+ 50us) client_negotiation.cc:240] Negotiation successful
1214 03:02:07.948563 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"spinlock_wait_cycles":21376,"thread_start_us":147,"threads_started":1}
I20251214 03:02:07.949141 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:07.945164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:40737 (local address 127.1.157.195:41381)
1214 03:02:07.945507 (+ 343us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:07.945511 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:07.945580 (+ 69us) server_negotiation.cc:408] Connection header received
1214 03:02:07.945749 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:07.945753 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:07.945805 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:07.945904 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:07.946413 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:07.947237 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:07.948414 (+ 1177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:07.948893 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:07.948923 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:07.948971 (+ 48us) server_negotiation.cc:300] Negotiation successful
1214 03:02:07.949021 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":118,"threads_started":1}
I20251214 03:02:07.949141 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:07.945261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:37733 (local address 127.1.157.194:39591)
1214 03:02:07.945497 (+ 236us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:07.945502 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:07.945580 (+ 78us) server_negotiation.cc:408] Connection header received
1214 03:02:07.945749 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:07.945753 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:07.945805 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:07.945904 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:07.946414 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:07.947204 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:07.948414 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:07.948889 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:07.948923 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:07.948971 (+ 48us) server_negotiation.cc:300] Negotiation successful
1214 03:02:07.949020 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
I20251214 03:02:07.949527 3713 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "922cd2aee60248898b1dab8ec200a56e" candidate_uuid: "9ddec69b8741438385f4a042ca559b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6464e90933414cd7802684d3e98fd4e4" is_pre_election: true
I20251214 03:02:07.949549 3857 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "922cd2aee60248898b1dab8ec200a56e" candidate_uuid: "9ddec69b8741438385f4a042ca559b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3080dfb9f744188845903601db970b" is_pre_election: true
I20251214 03:02:07.949653 3713 raft_consensus.cc:2468] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ddec69b8741438385f4a042ca559b2e in term 0.
I20251214 03:02:07.949662 3857 raft_consensus.cc:2468] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ddec69b8741438385f4a042ca559b2e in term 0.
I20251214 03:02:07.949971 3505 leader_election.cc:304] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6464e90933414cd7802684d3e98fd4e4, 9ddec69b8741438385f4a042ca559b2e; no voters:
I20251214 03:02:07.950161 4032 raft_consensus.cc:2804] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:07.950217 4032 raft_consensus.cc:493] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:07.950284 4032 raft_consensus.cc:3060] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:07.951138 4032 raft_consensus.cc:515] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:07.951465 4032 leader_election.cc:290] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [CANDIDATE]: Term 1 election: Requested vote from peers 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591), 5d3080dfb9f744188845903601db970b (127.1.157.195:41381)
I20251214 03:02:07.951879 3857 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "922cd2aee60248898b1dab8ec200a56e" candidate_uuid: "9ddec69b8741438385f4a042ca559b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3080dfb9f744188845903601db970b"
I20251214 03:02:07.951953 3713 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "922cd2aee60248898b1dab8ec200a56e" candidate_uuid: "9ddec69b8741438385f4a042ca559b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6464e90933414cd7802684d3e98fd4e4"
I20251214 03:02:07.952006 3857 raft_consensus.cc:3060] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:07.952061 3713 raft_consensus.cc:3060] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:07.952947 3713 raft_consensus.cc:2468] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddec69b8741438385f4a042ca559b2e in term 1.
I20251214 03:02:07.953009 3857 raft_consensus.cc:2468] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddec69b8741438385f4a042ca559b2e in term 1.
I20251214 03:02:07.953336 3507 leader_election.cc:304] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3080dfb9f744188845903601db970b, 9ddec69b8741438385f4a042ca559b2e; no voters:
I20251214 03:02:07.953532 4032 raft_consensus.cc:2804] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:07.953724 4032 raft_consensus.cc:697] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 1 LEADER]: Becoming Leader. State: Replica: 9ddec69b8741438385f4a042ca559b2e, State: Running, Role: LEADER
I20251214 03:02:07.953958 4032 consensus_queue.cc:237] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:07.956063 3409 catalog_manager.cc:5654] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ddec69b8741438385f4a042ca559b2e (127.1.157.193). New cstate: current_term: 1 leader_uuid: "9ddec69b8741438385f4a042ca559b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } health_report { overall_health: UNKNOWN } } }
I20251214 03:02:08.010577 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:08.007363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.1.157.193:43107)
1214 03:02:08.007653 (+ 290us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:08.007657 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:08.007668 (+ 11us) server_negotiation.cc:408] Connection header received
1214 03:02:08.007742 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:08.007744 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:08.007804 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:08.007902 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:08.008252 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:08.008761 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:08.009476 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:08.009654 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:08.009708 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:08.010185 (+ 477us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:08.010285 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:08.010373 (+ 88us) server_negotiation.cc:300] Negotiation successful
1214 03:02:08.010427 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
I20251214 03:02:08.019304 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:08.017104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:60241 (local address 127.1.157.195:41381)
1214 03:02:08.017249 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:08.017252 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:08.017278 (+ 26us) server_negotiation.cc:408] Connection header received
1214 03:02:08.017395 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:08.017397 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:08.017441 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:08.017510 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:08.017816 (+ 306us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:08.018262 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:08.018827 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:08.018944 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:08.018977 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:08.019037 (+ 60us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:08.019087 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:08.019163 (+ 76us) server_negotiation.cc:300] Negotiation successful
1214 03:02:08.019198 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251214 03:02:08.019320 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:08.016914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41381 (local address 127.1.157.193:60241)
1214 03:02:08.017218 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:08.017234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:08.017317 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:08.017515 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:08.017517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:08.017523 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:02:08.017739 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:08.017744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:08.018333 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:08.018336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:08.018752 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:08.018757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:08.018835 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:08.018867 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:02:08.019100 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:02:08.019104 (+ 4us) client_negotiation.cc:769] Sending connection context
1214 03:02:08.019147 (+ 43us) client_negotiation.cc:240] Negotiation successful
1214 03:02:08.019198 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
I20251214 03:02:08.025463 3713 raft_consensus.cc:1275] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Refusing update from remote peer 9ddec69b8741438385f4a042ca559b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:08.025463 3857 raft_consensus.cc:1275] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Refusing update from remote peer 9ddec69b8741438385f4a042ca559b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:08.025883 4032 consensus_queue.cc:1048] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:08.026053 4037 consensus_queue.cc:1048] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:08.041383 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:08.038735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:42477 (local address 127.1.157.193:43107)
1214 03:02:08.038843 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:08.038847 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:08.039010 (+ 163us) server_negotiation.cc:408] Connection header received
1214 03:02:08.039106 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:08.039109 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:08.039151 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:08.039215 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:08.039529 (+ 314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:08.040025 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:08.040695 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:08.040854 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:08.040916 (+ 62us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:08.040982 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:08.041041 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:08.041189 (+ 148us) server_negotiation.cc:300] Negotiation successful
1214 03:02:08.041263 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251214 03:02:08.041393 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:08.038687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:43107 (local address 127.1.157.195:42477)
1214 03:02:08.038928 (+ 241us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:08.038943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:08.039038 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:08.039226 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:08.039228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:08.039235 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:02:08.039451 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:08.039456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:08.040117 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:08.040120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:08.040581 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:08.040591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:08.040671 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:08.040699 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:02:08.041052 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:02:08.041058 (+ 6us) client_negotiation.cc:769] Sending connection context
1214 03:02:08.041121 (+ 63us) client_negotiation.cc:240] Negotiation successful
1214 03:02:08.041217 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20251214 03:02:08.063599 3412 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42966:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:08.063745 3412 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42966:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:08.065840 3412 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822: Sending DeleteTablet for 3 replicas of tablet 922cd2aee60248898b1dab8ec200a56e
I20251214 03:02:08.066426 3695 tablet_service.cc:1552] Processing DeleteTablet for tablet 922cd2aee60248898b1dab8ec200a56e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:08 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:49098
I20251214 03:02:08.066557 3836 tablet_service.cc:1552] Processing DeleteTablet for tablet 922cd2aee60248898b1dab8ec200a56e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:08 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:45186
I20251214 03:02:08.066642 4016 tablet_replica.cc:333] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4: stopping tablet replica
I20251214 03:02:08.066722 4017 tablet_replica.cc:333] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b: stopping tablet replica
I20251214 03:02:08.066752 4016 raft_consensus.cc:2243] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:08.066821 4017 raft_consensus.cc:2243] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:08.066860 4016 raft_consensus.cc:2272] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:08.066920 4017 raft_consensus.cc:2272] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:08.067116 3553 tablet_service.cc:1552] Processing DeleteTablet for tablet 922cd2aee60248898b1dab8ec200a56e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:08 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:54586
I20251214 03:02:08.067284 4015 tablet_replica.cc:333] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e: stopping tablet replica
I20251214 03:02:08.067377 4015 raft_consensus.cc:2243] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:02:08.067624 4017 ts_tablet_manager.cc:1916] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:08.067643 4015 raft_consensus.cc:2272] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:08.067581 3412 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:02:08.067991 3412 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:08.069247 4015 ts_tablet_manager.cc:1916] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:08.070098 4017 ts_tablet_manager.cc:1929] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:02:08.070183 4017 log.cc:1199] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/922cd2aee60248898b1dab8ec200a56e
I20251214 03:02:08.070407 4017 ts_tablet_manager.cc:1950] T 922cd2aee60248898b1dab8ec200a56e P 5d3080dfb9f744188845903601db970b: Deleting consensus metadata
I20251214 03:02:08.071142 3401 catalog_manager.cc:4985] TS 5d3080dfb9f744188845903601db970b (127.1.157.195:41381): tablet 922cd2aee60248898b1dab8ec200a56e (table test-table [id=eda7fad749ee437cb9d427e9965e26af]) successfully deleted
I20251214 03:02:08.071601 4015 ts_tablet_manager.cc:1929] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:02:08.071669 4015 log.cc:1199] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-0/wal/wals/922cd2aee60248898b1dab8ec200a56e
I20251214 03:02:08.071887 4015 ts_tablet_manager.cc:1950] T 922cd2aee60248898b1dab8ec200a56e P 9ddec69b8741438385f4a042ca559b2e: Deleting consensus metadata
I20251214 03:02:08.072603 3401 catalog_manager.cc:4985] TS 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107): tablet 922cd2aee60248898b1dab8ec200a56e (table test-table [id=eda7fad749ee437cb9d427e9965e26af]) successfully deleted
I20251214 03:02:08.072937 4016 ts_tablet_manager.cc:1916] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:08.073088 3553 tablet_service.cc:1505] Processing CreateTablet for tablet 47ee82d64fb74780baaef3ebdd9caf4e (DEFAULT_TABLE table=test-table [id=6fa770e46815459fbb6b4d35042bc6b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:08.073369 3553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ee82d64fb74780baaef3ebdd9caf4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:08.074180 3695 tablet_service.cc:1505] Processing CreateTablet for tablet 47ee82d64fb74780baaef3ebdd9caf4e (DEFAULT_TABLE table=test-table [id=6fa770e46815459fbb6b4d35042bc6b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:08.074397 3695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ee82d64fb74780baaef3ebdd9caf4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:08.075228 4016 ts_tablet_manager.cc:1929] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:02:08.075294 4016 log.cc:1199] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/922cd2aee60248898b1dab8ec200a56e
I20251214 03:02:08.075399 4026 tablet_bootstrap.cc:492] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e: Bootstrap starting.
I20251214 03:02:08.075531 4016 ts_tablet_manager.cc:1950] T 922cd2aee60248898b1dab8ec200a56e P 6464e90933414cd7802684d3e98fd4e4: Deleting consensus metadata
I20251214 03:02:08.076151 3399 catalog_manager.cc:4985] TS 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591): tablet 922cd2aee60248898b1dab8ec200a56e (table test-table [id=eda7fad749ee437cb9d427e9965e26af]) successfully deleted
I20251214 03:02:08.076390 4026 tablet_bootstrap.cc:654] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:08.076408 4027 tablet_bootstrap.cc:492] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4: Bootstrap starting.
I20251214 03:02:08.076900 3836 tablet_service.cc:1505] Processing CreateTablet for tablet 47ee82d64fb74780baaef3ebdd9caf4e (DEFAULT_TABLE table=test-table [id=6fa770e46815459fbb6b4d35042bc6b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:08.077144 3836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ee82d64fb74780baaef3ebdd9caf4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:08.077251 4027 tablet_bootstrap.cc:654] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:08.077538 4026 tablet_bootstrap.cc:492] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e: No bootstrap required, opened a new log
I20251214 03:02:08.077611 4026 ts_tablet_manager.cc:1403] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:08.078043 4026 raft_consensus.cc:359] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:08.078140 4026 raft_consensus.cc:385] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:08.078167 4026 raft_consensus.cc:740] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ddec69b8741438385f4a042ca559b2e, State: Initialized, Role: FOLLOWER
I20251214 03:02:08.078214 4027 tablet_bootstrap.cc:492] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4: No bootstrap required, opened a new log
I20251214 03:02:08.078286 4027 ts_tablet_manager.cc:1403] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251214 03:02:08.078290 4026 consensus_queue.cc:260] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:08.078547 4026 ts_tablet_manager.cc:1434] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:08.078660 4027 raft_consensus.cc:359] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:08.078753 4027 raft_consensus.cc:385] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:08.078780 4027 raft_consensus.cc:740] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6464e90933414cd7802684d3e98fd4e4, State: Initialized, Role: FOLLOWER
I20251214 03:02:08.078886 4027 consensus_queue.cc:260] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:08.079123 4027 ts_tablet_manager.cc:1434] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251214 03:02:08.079398 4028 tablet_bootstrap.cc:492] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b: Bootstrap starting.
I20251214 03:02:08.080515 4028 tablet_bootstrap.cc:654] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:08.081643 4028 tablet_bootstrap.cc:492] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b: No bootstrap required, opened a new log
I20251214 03:02:08.081743 4028 ts_tablet_manager.cc:1403] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:08.082202 4028 raft_consensus.cc:359] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:08.082306 4028 raft_consensus.cc:385] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:08.082340 4028 raft_consensus.cc:740] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3080dfb9f744188845903601db970b, State: Initialized, Role: FOLLOWER
I20251214 03:02:08.082468 4028 consensus_queue.cc:260] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:08.082718 4028 ts_tablet_manager.cc:1434] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:08.272306 4037 raft_consensus.cc:493] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:08.272478 4037 raft_consensus.cc:515] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:08.272935 4037 leader_election.cc:290] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591), 5d3080dfb9f744188845903601db970b (127.1.157.195:41381)
I20251214 03:02:08.273399 3857 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ee82d64fb74780baaef3ebdd9caf4e" candidate_uuid: "9ddec69b8741438385f4a042ca559b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3080dfb9f744188845903601db970b" is_pre_election: true
I20251214 03:02:08.273404 3713 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ee82d64fb74780baaef3ebdd9caf4e" candidate_uuid: "9ddec69b8741438385f4a042ca559b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6464e90933414cd7802684d3e98fd4e4" is_pre_election: true
I20251214 03:02:08.273528 3713 raft_consensus.cc:2468] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ddec69b8741438385f4a042ca559b2e in term 0.
I20251214 03:02:08.273535 3857 raft_consensus.cc:2468] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ddec69b8741438385f4a042ca559b2e in term 0.
I20251214 03:02:08.273830 3507 leader_election.cc:304] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3080dfb9f744188845903601db970b, 9ddec69b8741438385f4a042ca559b2e; no voters:
I20251214 03:02:08.274006 4037 raft_consensus.cc:2804] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:08.274070 4037 raft_consensus.cc:493] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:08.274106 4037 raft_consensus.cc:3060] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:08.275038 4037 raft_consensus.cc:515] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:08.275416 4037 leader_election.cc:290] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [CANDIDATE]: Term 1 election: Requested vote from peers 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591), 5d3080dfb9f744188845903601db970b (127.1.157.195:41381)
I20251214 03:02:08.275707 3713 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ee82d64fb74780baaef3ebdd9caf4e" candidate_uuid: "9ddec69b8741438385f4a042ca559b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6464e90933414cd7802684d3e98fd4e4"
I20251214 03:02:08.275754 3857 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ee82d64fb74780baaef3ebdd9caf4e" candidate_uuid: "9ddec69b8741438385f4a042ca559b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3080dfb9f744188845903601db970b"
I20251214 03:02:08.275825 3713 raft_consensus.cc:3060] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:08.275836 3857 raft_consensus.cc:3060] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:08.276799 3713 raft_consensus.cc:2468] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddec69b8741438385f4a042ca559b2e in term 1.
I20251214 03:02:08.276799 3857 raft_consensus.cc:2468] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddec69b8741438385f4a042ca559b2e in term 1.
I20251214 03:02:08.277065 3505 leader_election.cc:304] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6464e90933414cd7802684d3e98fd4e4, 9ddec69b8741438385f4a042ca559b2e; no voters:
I20251214 03:02:08.277245 4037 raft_consensus.cc:2804] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:08.277349 4037 raft_consensus.cc:697] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 1 LEADER]: Becoming Leader. State: Replica: 9ddec69b8741438385f4a042ca559b2e, State: Running, Role: LEADER
I20251214 03:02:08.277493 4037 consensus_queue.cc:237] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } }
I20251214 03:02:08.278908 3412 catalog_manager.cc:5654] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ddec69b8741438385f4a042ca559b2e (127.1.157.193). New cstate: current_term: 1 leader_uuid: "9ddec69b8741438385f4a042ca559b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddec69b8741438385f4a042ca559b2e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 } health_report { overall_health: UNKNOWN } } }
I20251214 03:02:08.333915 3713 raft_consensus.cc:1275] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Refusing update from remote peer 9ddec69b8741438385f4a042ca559b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:08.333915 3857 raft_consensus.cc:1275] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Refusing update from remote peer 9ddec69b8741438385f4a042ca559b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:08.334312 4037 consensus_queue.cc:1048] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6464e90933414cd7802684d3e98fd4e4" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:08.334460 4046 consensus_queue.cc:1048] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3080dfb9f744188845903601db970b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 03:02:08.345906 3996 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251214 03:02:09.895435 3412 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42966:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:09.895604 3412 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42966:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:09.898013 3412 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0970128eef274e7abb5c45b170a0a822: Sending DeleteTablet for 3 replicas of tablet 47ee82d64fb74780baaef3ebdd9caf4e
I20251214 03:02:09.898607 3553 tablet_service.cc:1552] Processing DeleteTablet for tablet 47ee82d64fb74780baaef3ebdd9caf4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:09 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:54586
I20251214 03:02:09.898667 3695 tablet_service.cc:1552] Processing DeleteTablet for tablet 47ee82d64fb74780baaef3ebdd9caf4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:09 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:49098
I20251214 03:02:09.898726 3836 tablet_service.cc:1552] Processing DeleteTablet for tablet 47ee82d64fb74780baaef3ebdd9caf4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:09 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:45186
I20251214 03:02:09.898960 4077 tablet_replica.cc:333] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e: stopping tablet replica
I20251214 03:02:09.899053 4078 tablet_replica.cc:333] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4: stopping tablet replica
I20251214 03:02:09.899082 4077 raft_consensus.cc:2243] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:02:09.899173 4078 raft_consensus.cc:2243] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:09.899276 4077 raft_consensus.cc:2272] T 47ee82d64fb74780baaef3ebdd9caf4e P 9ddec69b8741438385f4a042ca559b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:09.899286 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 3488
I20251214 03:02:09.899291 4078 raft_consensus.cc:2272] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:09.901675 4079 tablet_replica.cc:333] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b: stopping tablet replica
I20251214 03:02:09.901831 4079 raft_consensus.cc:2243] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:09.901950 4079 raft_consensus.cc:2272] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:09.902045 4078 ts_tablet_manager.cc:1916] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:09.902817 4079 ts_tablet_manager.cc:1916] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:09.904551 4078 ts_tablet_manager.cc:1929] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251214 03:02:09.904623 4078 log.cc:1199] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/47ee82d64fb74780baaef3ebdd9caf4e
I20251214 03:02:09.904707 4079 ts_tablet_manager.cc:1929] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251214 03:02:09.904773 4079 log.cc:1199] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/47ee82d64fb74780baaef3ebdd9caf4e
I20251214 03:02:09.904877 4078 ts_tablet_manager.cc:1950] T 47ee82d64fb74780baaef3ebdd9caf4e P 6464e90933414cd7802684d3e98fd4e4: Deleting consensus metadata
I20251214 03:02:09.905033 4079 ts_tablet_manager.cc:1950] T 47ee82d64fb74780baaef3ebdd9caf4e P 5d3080dfb9f744188845903601db970b: Deleting consensus metadata
I20251214 03:02:09.905632 3401 catalog_manager.cc:4985] TS 5d3080dfb9f744188845903601db970b (127.1.157.195:41381): tablet 47ee82d64fb74780baaef3ebdd9caf4e (table test-table [id=6fa770e46815459fbb6b4d35042bc6b0]) successfully deleted
I20251214 03:02:09.905593 3399 catalog_manager.cc:4985] TS 6464e90933414cd7802684d3e98fd4e4 (127.1.157.194:39591): tablet 47ee82d64fb74780baaef3ebdd9caf4e (table test-table [id=6fa770e46815459fbb6b4d35042bc6b0]) successfully deleted
W20251214 03:02:09.906545 3401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.157.193:43107 (error 108)
I20251214 03:02:09.906754 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 3630
I20251214 03:02:09.908174 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:09.907783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:43107 (local address 127.0.0.1:54616)
1214 03:02:09.908025 (+ 242us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:09.908086 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.193:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
W20251214 03:02:09.908300 3401 catalog_manager.cc:4712] TS 9ddec69b8741438385f4a042ca559b2e (127.1.157.193:43107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47ee82d64fb74780baaef3ebdd9caf4e: Network error: Client connection negotiation failed: client connection to 127.1.157.193:43107: connect: Connection refused (error 111)
I20251214 03:02:09.913007 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 3772
I20251214 03:02:09.918576 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 3382
2025-12-14T03:02:09Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7437 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:02:09 dist-test-slave-k7z7 krb5kdc[4084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:02:09 dist-test-slave-k7z7 krb5kdc[4084](info): set up 2 sockets
Dec 14 03:02:09 dist-test-slave-k7z7 krb5kdc[4084](info): commencing operation
krb5kdc: starting...
W20251214 03:02:11.757091 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.818s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:02:11 dist-test-slave-k7z7 krb5kdc[4084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:02:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:11.897459 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:40703
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:40703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:12.003784 4100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:12.004053 4100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:12.004138 4100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:12.009131 4100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:12.009478 4100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:12.009531 4100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:12.009567 4100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:12.009603 4100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:12.015700 4100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:40703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:40703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:12.016933 4100 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:12.017822 4100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:12.023509 4108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:12.023556 4105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:12.023566 4106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:12.023592 4100 server_base.cc:1047] running on GCE node
I20251214 03:02:12.024312 4100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:12.025216 4100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:12.026367 4100 hybrid_clock.cc:648] HybridClock initialized: now 1765681332026350 us; error 35 us; skew 500 ppm
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:12.028735 4100 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:02:12.029743 4100 webserver.cc:492] Webserver started at http://127.1.157.254:46157/ using document root <none> and password file <none>
I20251214 03:02:12.030234 4100 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:12.030280 4100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:12.030469 4100 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:12.032049 4100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "8d80183f0c694a64ba0c5771f21ccdd6"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "d45bbad83a21cb51d6a0eaf16450e49e"
server_key_iv: "e255ed0e3dd54f736ffa3e854bca290b"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.032533 4100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "8d80183f0c694a64ba0c5771f21ccdd6"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "d45bbad83a21cb51d6a0eaf16450e49e"
server_key_iv: "e255ed0e3dd54f736ffa3e854bca290b"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.035686 4100 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:02:12.037801 4115 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:12.038772 4100 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:12.038864 4100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "8d80183f0c694a64ba0c5771f21ccdd6"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "d45bbad83a21cb51d6a0eaf16450e49e"
server_key_iv: "e255ed0e3dd54f736ffa3e854bca290b"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.038955 4100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:12.053795 4100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:12.056474 4100 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:12.056622 4100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:12.063335 4100 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:40703
I20251214 03:02:12.063335 4167 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:40703 every 8 connection(s)
I20251214 03:02:12.064280 4100 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:12.066934 4168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:12.072279 4168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6: Bootstrap starting.
I20251214 03:02:12.072862 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 4100
I20251214 03:02:12.072957 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:12.073213 1655 external_mini_cluster.cc:1468] Setting key fe7190f2100be17bfc8ac0db4e7aceb4
I20251214 03:02:12.074779 4168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:12.075620 4168 log.cc:826] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:12.077672 4168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6: No bootstrap required, opened a new log
I20251214 03:02:12.080449 4168 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d80183f0c694a64ba0c5771f21ccdd6" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 40703 } }
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:12.080644 4168 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:12.080701 4168 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d80183f0c694a64ba0c5771f21ccdd6, State: Initialized, Role: FOLLOWER
I20251214 03:02:12.081144 4168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d80183f0c694a64ba0c5771f21ccdd6" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 40703 } }
I20251214 03:02:12.081327 4168 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:12.081410 4168 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:12.081511 4168 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:12.082425 4168 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d80183f0c694a64ba0c5771f21ccdd6" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 40703 } }
I20251214 03:02:12.082755 4168 leader_election.cc:304] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d80183f0c694a64ba0c5771f21ccdd6; no voters:
I20251214 03:02:12.083039 4168 leader_election.cc:290] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:12.083375 4173 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:12.083611 4173 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [term 1 LEADER]: Becoming Leader. State: Replica: 8d80183f0c694a64ba0c5771f21ccdd6, State: Running, Role: LEADER
I20251214 03:02:12.083942 4173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d80183f0c694a64ba0c5771f21ccdd6" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 40703 } }
I20251214 03:02:12.084311 4168 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:12.085470 4175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d80183f0c694a64ba0c5771f21ccdd6. Latest consensus state: current_term: 1 leader_uuid: "8d80183f0c694a64ba0c5771f21ccdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d80183f0c694a64ba0c5771f21ccdd6" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 40703 } } }
I20251214 03:02:12.085583 4175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:12.085902 4174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d80183f0c694a64ba0c5771f21ccdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d80183f0c694a64ba0c5771f21ccdd6" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 40703 } } }
I20251214 03:02:12.085969 4174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:12.085983 4182 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:12.086175 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:12.074309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.1.157.254:40703)
1214 03:02:12.074773 (+ 464us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:12.074786 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:12.074828 (+ 42us) server_negotiation.cc:408] Connection header received
1214 03:02:12.075525 (+ 697us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:12.075551 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:12.075920 (+ 369us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:12.076247 (+ 327us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:12.077105 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.078198 (+ 1093us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:12.078907 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.079199 (+ 292us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:12.081127 (+ 1928us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:12.081159 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:12.081203 (+ 44us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:12.081249 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:12.083053 (+ 1804us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:12.083557 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:12.083563 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:12.083569 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:12.083657 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:12.083895 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:12.083899 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:12.083901 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:12.084272 (+ 371us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:12.084481 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:12.085023 (+ 542us) server_negotiation.cc:300] Negotiation successful
1214 03:02:12.085321 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":122,"threads_started":1}
I20251214 03:02:12.089043 4182 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:12.094457 4182 catalog_manager.cc:1357] Generated new cluster ID: b902874c38f64c7fbd03426171879829
I20251214 03:02:12.094532 4182 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:12.120026 4182 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:12.121578 4182 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:12.138947 4182 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8d80183f0c694a64ba0c5771f21ccdd6: Generated new TSK 0
I20251214 03:02:12.139761 4182 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:12.193426 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:40703
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:12.299114 4196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:12.299322 4196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:12.299387 4196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:12.302600 4196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:12.302657 4196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:12.302752 4196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:12.306885 4196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:12.307960 4196 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:12.308753 4196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:12.314900 4202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:12.314904 4201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:12.315042 4196 server_base.cc:1047] running on GCE node
W20251214 03:02:12.315135 4204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:12.315464 4196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:12.315999 4196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:12.317155 4196 hybrid_clock.cc:648] HybridClock initialized: now 1765681332317134 us; error 38 us; skew 500 ppm
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:12.319584 4196 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:02:12.320569 4196 webserver.cc:492] Webserver started at http://127.1.157.193:35857/ using document root <none> and password file <none>
I20251214 03:02:12.321077 4196 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:12.321126 4196 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:12.321326 4196 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:12.322939 4196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "3350c82988ce463bbe8989cd5cd809aa"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "c86c8f96bb7b349329ffd2de1b3e76cc"
server_key_iv: "24dd7aefea28877c93054ffe8ea7adbd"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.323375 4196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "3350c82988ce463bbe8989cd5cd809aa"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "c86c8f96bb7b349329ffd2de1b3e76cc"
server_key_iv: "24dd7aefea28877c93054ffe8ea7adbd"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.326468 4196 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20251214 03:02:12.328405 4211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:12.329367 4196 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251214 03:02:12.329468 4196 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "3350c82988ce463bbe8989cd5cd809aa"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "c86c8f96bb7b349329ffd2de1b3e76cc"
server_key_iv: "24dd7aefea28877c93054ffe8ea7adbd"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.329541 4196 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:12.351137 4196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:12.354530 4196 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:12.354737 4196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:12.355319 4196 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:12.356248 4196 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:12.356302 4196 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:12.356360 4196 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:12.356398 4196 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:12.366328 4196 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:36141
I20251214 03:02:12.366346 4324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:36141 every 8 connection(s)
I20251214 03:02:12.367269 4196 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:02:12.368842 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 4196
I20251214 03:02:12.368938 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:02:12.369158 1655 external_mini_cluster.cc:1468] Setting key e246a5bc91511eb903d5f8f431145ce6
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:02:12.379652 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:12.368904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:51753 (local address 127.1.157.254:40703)
1214 03:02:12.369036 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:12.369041 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:12.369816 (+ 775us) server_negotiation.cc:408] Connection header received
1214 03:02:12.370773 (+ 957us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:12.370776 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:12.370820 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:12.370876 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:12.372784 (+ 1908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.373551 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:12.374223 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.374419 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:12.376563 (+ 2144us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:12.376586 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:12.376589 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:12.376618 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:12.378106 (+ 1488us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:12.378522 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:12.378525 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:12.378527 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:12.378564 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:12.378770 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:12.378772 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:12.378773 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:12.378924 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:12.378992 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:12.379425 (+ 433us) server_negotiation.cc:300] Negotiation successful
1214 03:02:12.379530 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
I20251214 03:02:12.380316 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:12.369245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:40703 (local address 127.1.157.193:51753)
1214 03:02:12.369667 (+ 422us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:12.369710 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:12.370565 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:12.371053 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:12.371063 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:12.371564 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:12.372643 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:12.372658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.373660 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:12.373663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:12.374104 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:12.374112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.374308 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:12.374855 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:12.374879 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:12.376407 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:12.378211 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:12.378215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:12.378224 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:12.378432 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:12.378626 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:12.378628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:12.378629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:12.378709 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:12.379005 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:12.379010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:12.379228 (+ 218us) client_negotiation.cc:769] Sending connection context
1214 03:02:12.379393 (+ 165us) client_negotiation.cc:240] Negotiation successful
1214 03:02:12.379605 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":106,"threads_started":1}
I20251214 03:02:12.381464 4325 heartbeater.cc:344] Connected to a master server at 127.1.157.254:40703
I20251214 03:02:12.381734 4325 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:12.382246 4325 heartbeater.cc:507] Master 127.1.157.254:40703 requested a full tablet report, sending...
I20251214 03:02:12.383673 4132 ts_manager.cc:194] Registered new tserver with Master: 3350c82988ce463bbe8989cd5cd809aa (127.1.157.193:36141)
I20251214 03:02:12.384778 4132 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:51753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:12.416714 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:40703
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:12.522763 4332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:12.522984 4332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:12.523044 4332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:12.526415 4332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:12.526482 4332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:12.526567 4332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:12.530906 4332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:12.532023 4332 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:12.532868 4332 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:12.539254 4337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:12.539327 4338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:12.539264 4340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:12.539558 4332 server_base.cc:1047] running on GCE node
I20251214 03:02:12.539875 4332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:12.540370 4332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:12.541508 4332 hybrid_clock.cc:648] HybridClock initialized: now 1765681332541493 us; error 29 us; skew 500 ppm
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:12.543889 4332 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:02:12.544874 4332 webserver.cc:492] Webserver started at http://127.1.157.194:33119/ using document root <none> and password file <none>
I20251214 03:02:12.545445 4332 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:12.545495 4332 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:12.545662 4332 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:12.547271 4332 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "c0cf2e1d929d49d795bcb94d5dffa870"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "2bcfe051a253da977ed245ea07cfac1d"
server_key_iv: "3ae074421f9825638bd5bc67cc418184"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.547703 4332 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "c0cf2e1d929d49d795bcb94d5dffa870"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "2bcfe051a253da977ed245ea07cfac1d"
server_key_iv: "3ae074421f9825638bd5bc67cc418184"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.550812 4332 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:02:12.552754 4347 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:12.553692 4332 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:12.553789 4332 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "c0cf2e1d929d49d795bcb94d5dffa870"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "2bcfe051a253da977ed245ea07cfac1d"
server_key_iv: "3ae074421f9825638bd5bc67cc418184"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.553867 4332 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:12.564121 4332 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:12.567273 4332 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:12.567453 4332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:12.567998 4332 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:12.568881 4332 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:12.568930 4332 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:12.568982 4332 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:12.569007 4332 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:12.577778 4332 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:36929
I20251214 03:02:12.577800 4460 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:36929 every 8 connection(s)
I20251214 03:02:12.578661 4332 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:02:12.582078 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 4332
I20251214 03:02:12.582166 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:02:12.582394 1655 external_mini_cluster.cc:1468] Setting key 01e5ca7b8879f0bd54f86fc02de58637
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
I20251214 03:02:12.591391 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:12.580245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:46533 (local address 127.1.157.254:40703)
1214 03:02:12.580380 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:12.580383 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:12.581043 (+ 660us) server_negotiation.cc:408] Connection header received
1214 03:02:12.581888 (+ 845us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:12.581892 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:12.581950 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:12.582023 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:12.583914 (+ 1891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.584402 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:12.585110 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.585319 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:12.588112 (+ 2793us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:12.588130 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:12.588133 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:12.588157 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:12.589453 (+ 1296us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:12.589990 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:12.589993 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:12.589995 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:12.590037 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:12.590333 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:12.590335 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:12.590337 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:12.590472 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:12.590555 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:12.591173 (+ 618us) server_negotiation.cc:300] Negotiation successful
1214 03:02:12.591274 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251214 03:02:12.592031 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:12.580513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:40703 (local address 127.1.157.194:46533)
1214 03:02:12.580909 (+ 396us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:12.580945 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:12.581702 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:12.582197 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:12.582208 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:12.582644 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:12.583785 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:12.583802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.584513 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:12.584517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:12.584959 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:12.584969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.585773 (+ 804us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:12.586378 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:12.586401 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:12.587980 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:12.589578 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:12.589585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:12.589596 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:12.589893 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:12.590116 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:12.590120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:12.590122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:12.590249 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:12.590587 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:12.590594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:12.590877 (+ 283us) client_negotiation.cc:769] Sending connection context
1214 03:02:12.591117 (+ 240us) client_negotiation.cc:240] Negotiation successful
1214 03:02:12.591400 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":106,"threads_started":1}
I20251214 03:02:12.593082 4461 heartbeater.cc:344] Connected to a master server at 127.1.157.254:40703
I20251214 03:02:12.593355 4461 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
I20251214 03:02:12.593923 4461 heartbeater.cc:507] Master 127.1.157.254:40703 requested a full tablet report, sending...
I20251214 03:02:12.595026 4132 ts_manager.cc:194] Registered new tserver with Master: c0cf2e1d929d49d795bcb94d5dffa870 (127.1.157.194:36929)
I20251214 03:02:12.595634 4132 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:46533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:12.628391 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:40703
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:12.731343 4468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:12.731563 4468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:12.731640 4468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:12.734946 4468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:12.735013 4468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:12.735107 4468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:12.739287 4468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:12.740303 4468 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:12.741096 4468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:12.747694 4474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:12.747687 4476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:12.747687 4473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:12.747987 4468 server_base.cc:1047] running on GCE node
I20251214 03:02:12.748394 4468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:12.748943 4468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:12.750083 4468 hybrid_clock.cc:648] HybridClock initialized: now 1765681332750056 us; error 45 us; skew 500 ppm
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:12.752744 4468 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:02:12.753890 4468 webserver.cc:492] Webserver started at http://127.1.157.195:44909/ using document root <none> and password file <none>
I20251214 03:02:12.754431 4468 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:12.754491 4468 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:12.754664 4468 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:12.756323 4468 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "eadcbd6b624e474d8d486f28ff477ede"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "bd419f092cef680274ce72ea5c9ac489"
server_key_iv: "9e0e0a273da0b7e7c0cc4d3e5e2baf70"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.756795 4468 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "eadcbd6b624e474d8d486f28ff477ede"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "bd419f092cef680274ce72ea5c9ac489"
server_key_iv: "9e0e0a273da0b7e7c0cc4d3e5e2baf70"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.760238 4468 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251214 03:02:12.762409 4483 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:12.763402 4468 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:12.763514 4468 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "eadcbd6b624e474d8d486f28ff477ede"
format_stamp: "Formatted at 2025-12-14 03:02:12 on dist-test-slave-k7z7"
server_key: "bd419f092cef680274ce72ea5c9ac489"
server_key_iv: "9e0e0a273da0b7e7c0cc4d3e5e2baf70"
server_key_version: "encryptionkey@0"
I20251214 03:02:12.763598 4468 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:12.776535 4468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:12.782634 4468 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:12.782795 4468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:12.783314 4468 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:12.784147 4468 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:12.784202 4468 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:12.784240 4468 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:12.784260 4468 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:12.792690 4468 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:46441
I20251214 03:02:12.792721 4596 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:46441 every 8 connection(s)
I20251214 03:02:12.793720 4468 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:12.803576 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 4468
I20251214 03:02:12.803683 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:02:12.803900 1655 external_mini_cluster.cc:1468] Setting key 976bb52306c542285ee458c076b0eea3
I20251214 03:02:12.806151 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:12.795415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:49229 (local address 127.1.157.254:40703)
1214 03:02:12.795568 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:12.795572 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:12.796397 (+ 825us) server_negotiation.cc:408] Connection header received
1214 03:02:12.797296 (+ 899us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:12.797299 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:12.797352 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:12.797425 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:12.799089 (+ 1664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.799629 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:12.800304 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.800458 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:12.802674 (+ 2216us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:12.802696 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:12.802699 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:12.802727 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:12.804139 (+ 1412us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:12.804604 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:12.804608 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:12.804611 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:12.804662 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:12.804988 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:12.804991 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:12.804993 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:12.805216 (+ 223us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:12.805342 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:12.805908 (+ 566us) server_negotiation.cc:300] Negotiation successful
1214 03:02:12.806031 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251214 03:02:12.806869 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:12.795726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:40703 (local address 127.1.157.195:49229)
1214 03:02:12.796256 (+ 530us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:12.796305 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:12.797063 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:12.797559 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:12.797567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:12.797951 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:12.798921 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:12.798937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.799766 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:12.799770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:12.800177 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:12.800185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:12.800376 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:12.800966 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:12.800992 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:12.802509 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:12.804259 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:12.804265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:12.804275 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:12.804517 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:12.804753 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:12.804756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:12.804759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:12.804892 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:12.805381 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:12.805390 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:12.805645 (+ 255us) client_negotiation.cc:769] Sending connection context
1214 03:02:12.805861 (+ 216us) client_negotiation.cc:240] Negotiation successful
1214 03:02:12.806121 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20251214 03:02:12.807857 4597 heartbeater.cc:344] Connected to a master server at 127.1.157.254:40703
I20251214 03:02:12.808092 4597 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:12.808614 4597 heartbeater.cc:507] Master 127.1.157.254:40703 requested a full tablet report, sending...
I20251214 03:02:12.809841 4132 ts_manager.cc:194] Registered new tserver with Master: eadcbd6b624e474d8d486f28ff477ede (127.1.157.195:46441)
I20251214 03:02:12.810417 4132 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:49229
I20251214 03:02:12.818826 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251214 03:02:12.830111 4171 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:12.828802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.1.157.254:40703)
1214 03:02:12.828941 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:12.828946 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:12.829038 (+ 92us) server_negotiation.cc:408] Connection header received
1214 03:02:12.829215 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:12.829218 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:12.829271 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:12.829339 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:12.829986 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56160: BlockingRecv error: recv got EOF from 127.0.0.1:56160 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20251214 03:02:12.831319 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 4196
I20251214 03:02:12.836215 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 4332
I20251214 03:02:12.840950 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 4468
I20251214 03:02:12.845572 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 4100
2025-12-14T03:02:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2927 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4609](info): set up 2 sockets
Dec 14 03:02:12 dist-test-slave-k7z7 krb5kdc[4609](info): commencing operation
krb5kdc: starting...
W20251214 03:02:14.689150 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.822s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:02:14 dist-test-slave-k7z7 krb5kdc[4609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:02:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:14.820544 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:38189
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:38189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:14.923560 4625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:14.923774 4625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:14.923825 4625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:14.926976 4625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:14.927035 4625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:14.927058 4625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:14.927075 4625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:14.927091 4625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:14.931299 4625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:38189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:38189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:14.932271 4625 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:14.933030 4625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:14.938239 4630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:14.938354 4633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:14.938410 4625 server_base.cc:1047] running on GCE node
W20251214 03:02:14.938239 4631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:14.938974 4625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:14.939852 4625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:14.940999 4625 hybrid_clock.cc:648] HybridClock initialized: now 1765681334940987 us; error 32 us; skew 500 ppm
Dec 14 03:02:14 dist-test-slave-k7z7 krb5kdc[4609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681334, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:14.943427 4625 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:02:14.944447 4625 webserver.cc:492] Webserver started at http://127.1.157.254:35805/ using document root <none> and password file <none>
I20251214 03:02:14.944952 4625 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:14.944998 4625 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:14.945218 4625 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:14.946859 4625 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "40bb50107ae64b359f9db40ff81fe412"
format_stamp: "Formatted at 2025-12-14 03:02:14 on dist-test-slave-k7z7"
server_key: "670e626529eb2ad0297cc69e10d23329"
server_key_iv: "26452f0b99b159538bb182925a860395"
server_key_version: "encryptionkey@0"
I20251214 03:02:14.947305 4625 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "40bb50107ae64b359f9db40ff81fe412"
format_stamp: "Formatted at 2025-12-14 03:02:14 on dist-test-slave-k7z7"
server_key: "670e626529eb2ad0297cc69e10d23329"
server_key_iv: "26452f0b99b159538bb182925a860395"
server_key_version: "encryptionkey@0"
I20251214 03:02:14.950709 4625 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:02:14.953080 4640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:14.954137 4625 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251214 03:02:14.954267 4625 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "40bb50107ae64b359f9db40ff81fe412"
format_stamp: "Formatted at 2025-12-14 03:02:14 on dist-test-slave-k7z7"
server_key: "670e626529eb2ad0297cc69e10d23329"
server_key_iv: "26452f0b99b159538bb182925a860395"
server_key_version: "encryptionkey@0"
I20251214 03:02:14.954365 4625 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:14.966445 4625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:14.969046 4625 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:14.969244 4625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:14.976032 4625 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:38189
I20251214 03:02:14.976033 4692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:38189 every 8 connection(s)
I20251214 03:02:14.976990 4625 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:14.979578 4693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:14.984705 4693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412: Bootstrap starting.
I20251214 03:02:14.986660 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 4625
I20251214 03:02:14.986776 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:14.987000 1655 external_mini_cluster.cc:1468] Setting key 4d24484f03c100fa0356ecb43af81903
I20251214 03:02:14.987077 4693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:14.987809 4693 log.cc:826] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:14.989765 4693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412: No bootstrap required, opened a new log
I20251214 03:02:14.992105 4693 raft_consensus.cc:359] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bb50107ae64b359f9db40ff81fe412" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38189 } }
I20251214 03:02:14.992281 4693 raft_consensus.cc:385] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:14.992655 4693 raft_consensus.cc:740] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bb50107ae64b359f9db40ff81fe412, State: Initialized, Role: FOLLOWER
I20251214 03:02:14.993086 4693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bb50107ae64b359f9db40ff81fe412" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38189 } }
I20251214 03:02:14.993233 4693 raft_consensus.cc:399] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:14.993294 4693 raft_consensus.cc:493] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:14.993378 4693 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [term 0 FOLLOWER]: Advancing to term 1
Dec 14 03:02:14 dist-test-slave-k7z7 krb5kdc[4609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:14.994351 4693 raft_consensus.cc:515] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bb50107ae64b359f9db40ff81fe412" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38189 } }
I20251214 03:02:14.994678 4693 leader_election.cc:304] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40bb50107ae64b359f9db40ff81fe412; no voters:
I20251214 03:02:14.994928 4693 leader_election.cc:290] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:14.995041 4698 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:14.995229 4698 raft_consensus.cc:697] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [term 1 LEADER]: Becoming Leader. State: Replica: 40bb50107ae64b359f9db40ff81fe412, State: Running, Role: LEADER
I20251214 03:02:14.995586 4698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bb50107ae64b359f9db40ff81fe412" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38189 } }
I20251214 03:02:14.996340 4693 sys_catalog.cc:565] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:14.997088 4700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40bb50107ae64b359f9db40ff81fe412. Latest consensus state: current_term: 1 leader_uuid: "40bb50107ae64b359f9db40ff81fe412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bb50107ae64b359f9db40ff81fe412" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38189 } } }
I20251214 03:02:14.997227 4700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:14.997407 4699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40bb50107ae64b359f9db40ff81fe412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bb50107ae64b359f9db40ff81fe412" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 38189 } } }
I20251214 03:02:14.997500 4699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:14.997668 4706 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:15.000248 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:14.988076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.1.157.254:38189)
1214 03:02:14.988454 (+ 378us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:14.988466 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:14.988510 (+ 44us) server_negotiation.cc:408] Connection header received
1214 03:02:14.989100 (+ 590us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:14.989118 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:14.989456 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:14.989753 (+ 297us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:14.990710 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:14.991477 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:14.992158 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:14.992439 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:14.994890 (+ 2451us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:14.994911 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:14.994925 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:14.994968 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:14.997486 (+ 2518us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:14.997960 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:14.997965 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:14.997970 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:14.998056 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:14.998323 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:14.998327 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:14.998329 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:14.998736 (+ 407us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:14.998936 (+ 200us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:14.999256 (+ 320us) server_negotiation.cc:300] Negotiation successful
1214 03:02:14.999518 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
I20251214 03:02:15.000711 4706 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:15.005780 4706 catalog_manager.cc:1357] Generated new cluster ID: 672da6177f8d4cb39b697020353aea33
I20251214 03:02:15.005856 4706 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:15.016747 4706 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:15.017735 4706 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:15.027513 4706 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 40bb50107ae64b359f9db40ff81fe412: Generated new TSK 0
I20251214 03:02:15.028517 4706 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:15.086349 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:38189
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:15.206718 4721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:15.206916 4721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:15.206980 4721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:15.210176 4721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:15.210240 4721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:15.210325 4721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:15.214484 4721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:38189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:15.215493 4721 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:15.216281 4721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:15.222656 4726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:15.222651 4729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:15.222648 4727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:15.223301 4721 server_base.cc:1047] running on GCE node
I20251214 03:02:15.223665 4721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:15.224188 4721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:15.225349 4721 hybrid_clock.cc:648] HybridClock initialized: now 1765681335225338 us; error 27 us; skew 500 ppm
Dec 14 03:02:15 dist-test-slave-k7z7 krb5kdc[4609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:15.227757 4721 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:02:15.228719 4721 webserver.cc:492] Webserver started at http://127.1.157.193:41085/ using document root <none> and password file <none>
I20251214 03:02:15.229275 4721 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:15.229328 4721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:15.229498 4721 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:15.231115 4721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "407782bafbee4b07be33722223b0be36"
format_stamp: "Formatted at 2025-12-14 03:02:15 on dist-test-slave-k7z7"
server_key: "84f43359527c8355f4a5ceb18072f5eb"
server_key_iv: "4248f07f120f457d58b1b7f62a670f62"
server_key_version: "encryptionkey@0"
I20251214 03:02:15.231541 4721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "407782bafbee4b07be33722223b0be36"
format_stamp: "Formatted at 2025-12-14 03:02:15 on dist-test-slave-k7z7"
server_key: "84f43359527c8355f4a5ceb18072f5eb"
server_key_iv: "4248f07f120f457d58b1b7f62a670f62"
server_key_version: "encryptionkey@0"
I20251214 03:02:15.234622 4721 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251214 03:02:15.236593 4736 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:15.237550 4721 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:15.237643 4721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "407782bafbee4b07be33722223b0be36"
format_stamp: "Formatted at 2025-12-14 03:02:15 on dist-test-slave-k7z7"
server_key: "84f43359527c8355f4a5ceb18072f5eb"
server_key_iv: "4248f07f120f457d58b1b7f62a670f62"
server_key_version: "encryptionkey@0"
I20251214 03:02:15.237720 4721 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:15.269901 4721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:15.272450 4721 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:15.272617 4721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:15.273134 4721 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:15.274008 4721 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:15.274056 4721 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:15.274107 4721 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:15.274135 4721 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:15.282783 4721 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:36667
I20251214 03:02:15.282815 4849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:36667 every 8 connection(s)
I20251214 03:02:15.283674 4721 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
Dec 14 03:02:15 dist-test-slave-k7z7 krb5kdc[4609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:15.292320 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 4721
I20251214 03:02:15.292418 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:02:15.292682 1655 external_mini_cluster.cc:1468] Setting key aede19737856a97fde8fe49baa58dfc1
I20251214 03:02:15.295610 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:15.285329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:35793 (local address 127.1.157.254:38189)
1214 03:02:15.285467 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:15.285472 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:15.286130 (+ 658us) server_negotiation.cc:408] Connection header received
1214 03:02:15.286955 (+ 825us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:15.286959 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:15.287016 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:15.287087 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:15.288683 (+ 1596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.289224 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:15.289834 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.290018 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:15.292201 (+ 2183us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:15.292222 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:15.292225 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:15.292250 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:15.293819 (+ 1569us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:15.294329 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:15.294332 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:15.294334 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:15.294375 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:15.294659 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:15.294661 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:15.294662 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:15.294798 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:15.294864 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:15.295385 (+ 521us) server_negotiation.cc:300] Negotiation successful
1214 03:02:15.295488 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251214 03:02:15.296309 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:15.285565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:38189 (local address 127.1.157.193:35793)
1214 03:02:15.285986 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:15.286020 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:15.286767 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:15.287269 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:15.287276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:15.287639 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:15.288529 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:15.288543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.289327 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:15.289330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:15.289730 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:15.289737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.289937 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:15.290571 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:15.290602 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:15.292060 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:15.293930 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:15.293936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:15.293947 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:15.294240 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:15.294464 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:15.294467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:15.294469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:15.294587 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:15.294882 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:15.294888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:15.295159 (+ 271us) client_negotiation.cc:769] Sending connection context
1214 03:02:15.295369 (+ 210us) client_negotiation.cc:240] Negotiation successful
1214 03:02:15.295639 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":93,"threads_started":1}
I20251214 03:02:15.297500 4850 heartbeater.cc:344] Connected to a master server at 127.1.157.254:38189
I20251214 03:02:15.297724 4850 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:15.298215 4850 heartbeater.cc:507] Master 127.1.157.254:38189 requested a full tablet report, sending...
I20251214 03:02:15.299691 4657 ts_manager.cc:194] Registered new tserver with Master: 407782bafbee4b07be33722223b0be36 (127.1.157.193:36667)
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:02:15.301312 4657 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:35793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:15.339666 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:38189
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:15.443856 4857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:15.444114 4857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:15.444191 4857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:15.448359 4857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:15.448427 4857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:15.448524 4857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:15.452842 4857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:38189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:15.453994 4857 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:15.454828 4857 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:15.461585 4862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:15.461587 4863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:15.461964 4857 server_base.cc:1047] running on GCE node
W20251214 03:02:15.462098 4865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:15.462466 4857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:15.463063 4857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:15.464478 4857 hybrid_clock.cc:648] HybridClock initialized: now 1765681335464451 us; error 45 us; skew 500 ppm
Dec 14 03:02:15 dist-test-slave-k7z7 krb5kdc[4609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:15.468091 4857 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:02:15.469314 4857 webserver.cc:492] Webserver started at http://127.1.157.194:40859/ using document root <none> and password file <none>
I20251214 03:02:15.469854 4857 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:15.469902 4857 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:15.470053 4857 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:15.471807 4857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "46689ae6cc884937b868487880cfbf1f"
format_stamp: "Formatted at 2025-12-14 03:02:15 on dist-test-slave-k7z7"
server_key: "a1436ced1f60aeecd02bd5f26026b550"
server_key_iv: "f381187688c1f67d77cfd44e128fd94f"
server_key_version: "encryptionkey@0"
I20251214 03:02:15.472270 4857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "46689ae6cc884937b868487880cfbf1f"
format_stamp: "Formatted at 2025-12-14 03:02:15 on dist-test-slave-k7z7"
server_key: "a1436ced1f60aeecd02bd5f26026b550"
server_key_iv: "f381187688c1f67d77cfd44e128fd94f"
server_key_version: "encryptionkey@0"
I20251214 03:02:15.475932 4857 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251214 03:02:15.478449 4872 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:15.479668 4857 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:15.479782 4857 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "46689ae6cc884937b868487880cfbf1f"
format_stamp: "Formatted at 2025-12-14 03:02:15 on dist-test-slave-k7z7"
server_key: "a1436ced1f60aeecd02bd5f26026b550"
server_key_iv: "f381187688c1f67d77cfd44e128fd94f"
server_key_version: "encryptionkey@0"
I20251214 03:02:15.479874 4857 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:15.501495 4857 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:15.504063 4857 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:15.504212 4857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:15.504774 4857 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:15.505657 4857 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:15.505705 4857 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:15.505743 4857 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:15.505764 4857 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:15.514050 4857 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:42667
I20251214 03:02:15.514086 4985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:42667 every 8 connection(s)
I20251214 03:02:15.514961 4857 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:02:15.515542 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 4857
I20251214 03:02:15.515640 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:02:15.515832 1655 external_mini_cluster.cc:1468] Setting key 8b6946c7354a84c6fa01ffd84a0c9f7a
Dec 14 03:02:15 dist-test-slave-k7z7 krb5kdc[4609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
I20251214 03:02:15.528275 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:15.516754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:54889 (local address 127.1.157.254:38189)
1214 03:02:15.516887 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:15.516892 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:15.517776 (+ 884us) server_negotiation.cc:408] Connection header received
1214 03:02:15.518648 (+ 872us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:15.518653 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:15.518709 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:15.518781 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:15.520794 (+ 2013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.521327 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:15.522169 (+ 842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.522339 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:15.524779 (+ 2440us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:15.524796 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:15.524798 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:15.524823 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:15.526226 (+ 1403us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:15.526776 (+ 550us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:15.526779 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:15.526780 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:15.526835 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:15.527193 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:15.527196 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:15.527197 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:15.527380 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:15.527482 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:15.527984 (+ 502us) server_negotiation.cc:300] Negotiation successful
1214 03:02:15.528105 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251214 03:02:15.528818 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:15.517042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:38189 (local address 127.1.157.194:54889)
1214 03:02:15.517632 (+ 590us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:15.517672 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:15.518455 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:15.518943 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:15.518953 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:15.519487 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:15.520644 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:15.520663 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.521437 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:15.521441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:15.522043 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:15.522054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.522283 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:15.522866 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:15.522898 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:15.524576 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:15.526365 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:15.526371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:15.526383 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:15.526654 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:15.526932 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:15.526935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:15.526936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:15.527055 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:15.527488 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:15.527494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:15.527748 (+ 254us) client_negotiation.cc:769] Sending connection context
1214 03:02:15.527916 (+ 168us) client_negotiation.cc:240] Negotiation successful
1214 03:02:15.528122 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":127,"threads_started":1}
I20251214 03:02:15.529845 4986 heartbeater.cc:344] Connected to a master server at 127.1.157.254:38189
I20251214 03:02:15.530077 4986 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:15.530591 4986 heartbeater.cc:507] Master 127.1.157.254:38189 requested a full tablet report, sending...
I20251214 03:02:15.531497 4657 ts_manager.cc:194] Registered new tserver with Master: 46689ae6cc884937b868487880cfbf1f (127.1.157.194:42667)
I20251214 03:02:15.532016 4657 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:54889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:15.560464 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:38189
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:15.662412 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:15.662663 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:15.662748 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:15.666288 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:15.666389 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:15.666507 4993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:15.671015 4993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:38189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:15.672073 4993 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:15.672881 4993 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:15.679471 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:15.679519 4999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:15.679490 5001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:15.679771 4993 server_base.cc:1047] running on GCE node
I20251214 03:02:15.680123 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:15.680682 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:15.681820 4993 hybrid_clock.cc:648] HybridClock initialized: now 1765681335681808 us; error 31 us; skew 500 ppm
Dec 14 03:02:15 dist-test-slave-k7z7 krb5kdc[4609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:15.684441 4993 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:02:15.685587 4993 webserver.cc:492] Webserver started at http://127.1.157.195:35513/ using document root <none> and password file <none>
I20251214 03:02:15.686100 4993 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:15.686144 4993 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:15.686297 4993 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:15.687932 4993 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "3ec148e2d92d41a480c77f9252a4124f"
format_stamp: "Formatted at 2025-12-14 03:02:15 on dist-test-slave-k7z7"
server_key: "e117fc5e0643d1071ae8a1694b4fe23b"
server_key_iv: "32507b466009b2ed44991c850d182183"
server_key_version: "encryptionkey@0"
I20251214 03:02:15.688344 4993 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "3ec148e2d92d41a480c77f9252a4124f"
format_stamp: "Formatted at 2025-12-14 03:02:15 on dist-test-slave-k7z7"
server_key: "e117fc5e0643d1071ae8a1694b4fe23b"
server_key_iv: "32507b466009b2ed44991c850d182183"
server_key_version: "encryptionkey@0"
I20251214 03:02:15.691573 4993 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251214 03:02:15.693586 5008 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:15.694511 4993 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:15.694608 4993 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "3ec148e2d92d41a480c77f9252a4124f"
format_stamp: "Formatted at 2025-12-14 03:02:15 on dist-test-slave-k7z7"
server_key: "e117fc5e0643d1071ae8a1694b4fe23b"
server_key_iv: "32507b466009b2ed44991c850d182183"
server_key_version: "encryptionkey@0"
I20251214 03:02:15.694686 4993 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:15.717403 4993 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:15.719893 4993 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:15.720057 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:15.720577 4993 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:15.721423 4993 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:15.721480 4993 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:15.721532 4993 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:15.721556 4993 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:15.730310 4993 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:40079
I20251214 03:02:15.730342 5121 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:40079 every 8 connection(s)
I20251214 03:02:15.731256 4993 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:02:15.735597 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 4993
I20251214 03:02:15.735725 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:02:15.735986 1655 external_mini_cluster.cc:1468] Setting key cb3dd6742c69fb2d30c28b436165c811
Dec 14 03:02:15 dist-test-slave-k7z7 krb5kdc[4609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:15.743395 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:15.732930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:50039 (local address 127.1.157.254:38189)
1214 03:02:15.733091 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:15.733096 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:15.733821 (+ 725us) server_negotiation.cc:408] Connection header received
1214 03:02:15.734621 (+ 800us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:15.734624 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:15.734674 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:15.734747 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:15.736733 (+ 1986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.737432 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:15.738084 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.738273 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:15.740390 (+ 2117us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:15.740407 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:15.740410 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:15.740437 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:15.741809 (+ 1372us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:15.742258 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:15.742261 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:15.742262 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:15.742305 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:15.742524 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:15.742526 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:15.742527 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:15.742670 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:15.742738 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:15.743163 (+ 425us) server_negotiation.cc:300] Negotiation successful
1214 03:02:15.743275 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251214 03:02:15.744045 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:15.733253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:38189 (local address 127.1.157.195:50039)
1214 03:02:15.733680 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:15.733716 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:15.734430 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:15.734888 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:15.734896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:15.735296 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:15.736503 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:15.736523 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.737561 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:15.737564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:15.737977 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:15.737986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:15.738202 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:15.738805 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:15.738829 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:15.740223 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:15.741911 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:15.741916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:15.741925 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:15.742158 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:15.742377 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:15.742379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:15.742380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:15.742461 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:15.742742 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:15.742747 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:15.742973 (+ 226us) client_negotiation.cc:769] Sending connection context
1214 03:02:15.743141 (+ 168us) client_negotiation.cc:240] Negotiation successful
1214 03:02:15.743371 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
I20251214 03:02:15.745086 5122 heartbeater.cc:344] Connected to a master server at 127.1.157.254:38189
I20251214 03:02:15.745352 5122 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:15.745857 5122 heartbeater.cc:507] Master 127.1.157.254:38189 requested a full tablet report, sending...
I20251214 03:02:15.746809 4657 ts_manager.cc:194] Registered new tserver with Master: 3ec148e2d92d41a480c77f9252a4124f (127.1.157.195:40079)
I20251214 03:02:15.747354 4657 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:50039
I20251214 03:02:15.749841 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251214 03:02:15.874737 4697 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:15.871261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.1.157.254:38189)
1214 03:02:15.871429 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:15.871433 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:15.872324 (+ 891us) server_negotiation.cc:408] Connection header received
1214 03:02:15.873291 (+ 967us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:15.873295 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:15.873359 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:15.873443 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:15.874605 (+ 1162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35312: BlockingRecv error: recv got EOF from 127.0.0.1:35312 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20251214 03:02:15.875874 4697 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:15.875195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.1.157.254:38189)
1214 03:02:15.875301 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:15.875304 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:15.875318 (+ 14us) server_negotiation.cc:408] Connection header received
1214 03:02:15.875374 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:15.875377 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:15.875418 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:15.875482 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:15.875775 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35328: BlockingRecv error: recv got EOF from 127.0.0.1:35328 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20251214 03:02:15.876781 4697 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:15.876119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.1.157.254:38189)
1214 03:02:15.876269 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:15.876272 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:15.876283 (+ 11us) server_negotiation.cc:408] Connection header received
1214 03:02:15.876310 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:15.876313 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:15.876346 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:15.876405 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:15.876695 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35336: BlockingRecv error: recv got EOF from 127.0.0.1:35336 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20251214 03:02:15.880148 4697 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:15.879557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.1.157.254:38189)
1214 03:02:15.879655 (+ 98us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:15.879658 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:15.879669 (+ 11us) server_negotiation.cc:408] Connection header received
1214 03:02:15.879697 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:15.879699 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:15.879731 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:15.879794 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:15.880065 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35350: BlockingRecv error: recv got EOF from 127.0.0.1:35350 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20251214 03:02:15.937526 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 4721
I20251214 03:02:15.942512 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 4857
I20251214 03:02:15.947332 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 4993
I20251214 03:02:15.951931 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 4625
2025-12-14T03:02:15Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3106 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:02:16 dist-test-slave-k7z7 krb5kdc[5145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:02:16 dist-test-slave-k7z7 krb5kdc[5145](info): set up 2 sockets
Dec 14 03:02:16 dist-test-slave-k7z7 krb5kdc[5145](info): commencing operation
krb5kdc: starting...
W20251214 03:02:17.836654 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.863s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20251214 03:02:18.090409 1655 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.254s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20251214 03:02:18.372265 1655 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.282s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20251214 03:02:18.485011 1655 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.113s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/test-user.keytab.
W20251214 03:02:19.313127 1655 mini_kdc.cc:290] Time spent creating keytab for test-user: real 0.828s user 0.000s sys 0.002s
Dec 14 03:02:19 dist-test-slave-k7z7 krb5kdc[5145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:02:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/kudu.keytab.
W20251214 03:02:20.544142 1655 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.1.157.254: real 1.212s user 0.001s sys 0.002s
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:20.587977 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:35759
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:38201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:35759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:20.700246 5161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:20.700498 5161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:20.700578 5161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:20.703985 5161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:20.704051 5161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:20.704082 5161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:20.704108 5161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:20.704144 5161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:20.708415 5161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:38201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:35759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:35759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:20.709703 5161 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:20.710594 5161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:20.716204 5166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:20.716195 5169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:20.716409 5161 server_base.cc:1047] running on GCE node
W20251214 03:02:20.716198 5167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:20.716970 5161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:20.717918 5161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:20.719064 5161 hybrid_clock.cc:648] HybridClock initialized: now 1765681340719045 us; error 34 us; skew 500 ppm
Dec 14 03:02:20 dist-test-slave-k7z7 krb5kdc[5145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681340, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:20.721477 5161 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:02:20.722482 5161 webserver.cc:492] Webserver started at http://127.1.157.254:46193/ using document root <none> and password file <none>
I20251214 03:02:20.723006 5161 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:20.723048 5161 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:20.723203 5161 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:20.724943 5161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "57ed37bf440d42b5ba44c69cc023d137"
format_stamp: "Formatted at 2025-12-14 03:02:20 on dist-test-slave-k7z7"
server_key: "e25d74609fd5fbbd9d2479be93d42292"
server_key_iv: "fc91f81ba71363e8ee663833eb37076b"
server_key_version: "encryptionkey@0"
I20251214 03:02:20.725459 5161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "57ed37bf440d42b5ba44c69cc023d137"
format_stamp: "Formatted at 2025-12-14 03:02:20 on dist-test-slave-k7z7"
server_key: "e25d74609fd5fbbd9d2479be93d42292"
server_key_iv: "fc91f81ba71363e8ee663833eb37076b"
server_key_version: "encryptionkey@0"
I20251214 03:02:20.729061 5161 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20251214 03:02:20.731705 5176 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:20.733036 5161 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251214 03:02:20.733211 5161 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "57ed37bf440d42b5ba44c69cc023d137"
format_stamp: "Formatted at 2025-12-14 03:02:20 on dist-test-slave-k7z7"
server_key: "e25d74609fd5fbbd9d2479be93d42292"
server_key_iv: "fc91f81ba71363e8ee663833eb37076b"
server_key_version: "encryptionkey@0"
I20251214 03:02:20.733327 5161 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:20.749851 5161 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:20.753772 5161 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:20.753962 5161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:20.762496 5161 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:35759
I20251214 03:02:20.762567 5228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:35759 every 8 connection(s)
I20251214 03:02:20.763648 5161 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:20.767390 5229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:20.773435 5229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137: Bootstrap starting.
I20251214 03:02:20.773936 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 5161
I20251214 03:02:20.774057 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:20.774310 1655 external_mini_cluster.cc:1468] Setting key c8775e4ab5ffd197b70e5394b9fe08b8
I20251214 03:02:20.776479 5229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:20.777312 5229 log.cc:826] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:20.779403 5229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137: No bootstrap required, opened a new log
Dec 14 03:02:20 dist-test-slave-k7z7 krb5kdc[5145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:20.782768 5229 raft_consensus.cc:359] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed37bf440d42b5ba44c69cc023d137" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35759 } }
I20251214 03:02:20.782984 5229 raft_consensus.cc:385] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:20.783041 5229 raft_consensus.cc:740] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ed37bf440d42b5ba44c69cc023d137, State: Initialized, Role: FOLLOWER
I20251214 03:02:20.783540 5229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed37bf440d42b5ba44c69cc023d137" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35759 } }
I20251214 03:02:20.783674 5229 raft_consensus.cc:399] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:20.783756 5229 raft_consensus.cc:493] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:20.783879 5229 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:20.784758 5229 raft_consensus.cc:515] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed37bf440d42b5ba44c69cc023d137" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35759 } }
I20251214 03:02:20.785215 5229 leader_election.cc:304] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57ed37bf440d42b5ba44c69cc023d137; no voters:
I20251214 03:02:20.785573 5229 leader_election.cc:290] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:20.785749 5234 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:20.786370 5234 raft_consensus.cc:697] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [term 1 LEADER]: Becoming Leader. State: Replica: 57ed37bf440d42b5ba44c69cc023d137, State: Running, Role: LEADER
I20251214 03:02:20.786736 5234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed37bf440d42b5ba44c69cc023d137" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35759 } }
I20251214 03:02:20.787036 5229 sys_catalog.cc:565] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:20.788070 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:20.775537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.1.157.254:35759)
1214 03:02:20.775967 (+ 430us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:20.775977 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:20.776013 (+ 36us) server_negotiation.cc:408] Connection header received
1214 03:02:20.776629 (+ 616us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:20.776651 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:20.776979 (+ 328us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:20.777347 (+ 368us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:20.778287 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:20.779047 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:20.779886 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:20.780110 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:20.782599 (+ 2489us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:20.782621 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:20.782633 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:20.782671 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:20.784629 (+ 1958us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:20.785101 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:20.785107 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:20.785112 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:20.785238 (+ 126us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:20.785494 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:20.785496 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:20.785498 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:20.786090 (+ 592us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:20.786307 (+ 217us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:20.786959 (+ 652us) server_negotiation.cc:300] Negotiation successful
1214 03:02:20.787242 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
I20251214 03:02:20.788249 5236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57ed37bf440d42b5ba44c69cc023d137. Latest consensus state: current_term: 1 leader_uuid: "57ed37bf440d42b5ba44c69cc023d137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed37bf440d42b5ba44c69cc023d137" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35759 } } }
I20251214 03:02:20.788354 5236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:20.788708 5243 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:20.789130 5235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57ed37bf440d42b5ba44c69cc023d137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed37bf440d42b5ba44c69cc023d137" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35759 } } }
I20251214 03:02:20.789309 5235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:20.791374 1655 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251214 03:02:20.792032 5243 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:20.798288 5243 catalog_manager.cc:1357] Generated new cluster ID: e02e35186ada4cb8a20755430b7bb59a
I20251214 03:02:20.798362 5243 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:20.812152 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:20.807201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.1.157.254:35759)
1214 03:02:20.807333 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:20.807337 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:20.807624 (+ 287us) server_negotiation.cc:408] Connection header received
1214 03:02:20.807747 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:20.807751 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:20.807816 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:20.807909 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:20.808677 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:20.809472 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:20.810299 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:20.810514 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:20.811018 (+ 504us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:20.811043 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:20.811047 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:20.811672 (+ 625us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:20.811715 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:20.811736 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:20.811833 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:20.811919 (+ 86us) server_negotiation.cc:300] Negotiation successful
1214 03:02:20.811993 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251214 03:02:20.815754 5243 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:20.817230 5243 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:20.825287 5243 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 57ed37bf440d42b5ba44c69cc023d137: Generated new TSK 0
I20251214 03:02:20.826081 5243 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 03:02:20.833266 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 5161
2025-12-14T03:02:20Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (4879 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:02:20 dist-test-slave-k7z7 krb5kdc[5262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:02:20 dist-test-slave-k7z7 krb5kdc[5262](info): set up 2 sockets
Dec 14 03:02:20 dist-test-slave-k7z7 krb5kdc[5262](info): commencing operation
krb5kdc: starting...
W20251214 03:02:22.693714 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.841s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:02:22 dist-test-slave-k7z7 krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:02:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:22.840131 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:33897
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:22.951545 5278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:22.951815 5278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:22.951893 5278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:22.955531 5278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:22.955622 5278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:22.955658 5278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:22.955682 5278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:22.955708 5278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:22.960355 5278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:22.961594 5278 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:22.962486 5278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:22.968411 5283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:22.968429 5286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:22.968493 5278 server_base.cc:1047] running on GCE node
W20251214 03:02:22.968411 5284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:22.969118 5278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:22.970152 5278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:22.971346 5278 hybrid_clock.cc:648] HybridClock initialized: now 1765681342971324 us; error 40 us; skew 500 ppm
Dec 14 03:02:22 dist-test-slave-k7z7 krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681342, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:22.974162 5278 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:02:22.975281 5278 webserver.cc:492] Webserver started at http://127.1.157.254:44761/ using document root <none> and password file <none>
I20251214 03:02:22.975805 5278 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:22.975854 5278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:22.976022 5278 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:22.977726 5278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "05f9c189dbbe46de8279b2fe462ccadd"
format_stamp: "Formatted at 2025-12-14 03:02:22 on dist-test-slave-k7z7"
server_key: "ea35a5df491851ded0eadc7722dcad91"
server_key_iv: "0fa58d3496485e75673607baad532406"
server_key_version: "encryptionkey@0"
I20251214 03:02:22.978166 5278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "05f9c189dbbe46de8279b2fe462ccadd"
format_stamp: "Formatted at 2025-12-14 03:02:22 on dist-test-slave-k7z7"
server_key: "ea35a5df491851ded0eadc7722dcad91"
server_key_iv: "0fa58d3496485e75673607baad532406"
server_key_version: "encryptionkey@0"
I20251214 03:02:22.981467 5278 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:02:22.983632 5293 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:22.984709 5278 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:22.984827 5278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "05f9c189dbbe46de8279b2fe462ccadd"
format_stamp: "Formatted at 2025-12-14 03:02:22 on dist-test-slave-k7z7"
server_key: "ea35a5df491851ded0eadc7722dcad91"
server_key_iv: "0fa58d3496485e75673607baad532406"
server_key_version: "encryptionkey@0"
I20251214 03:02:22.984922 5278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:23.009007 5278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:23.012087 5278 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:23.012277 5278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:23.019822 5278 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:33897
I20251214 03:02:23.019826 5345 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:33897 every 8 connection(s)
I20251214 03:02:23.020814 5278 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:23.023598 5346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:23.026096 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 5278
I20251214 03:02:23.026232 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:23.026458 1655 external_mini_cluster.cc:1468] Setting key c01f8ff563327bf4fac0f65d08f687bb
I20251214 03:02:23.029397 5346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd: Bootstrap starting.
I20251214 03:02:23.032212 5346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:23.033077 5346 log.cc:826] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd: Log is configured to *not* fsync() on all Append() calls
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:23.035197 5346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd: No bootstrap required, opened a new log
I20251214 03:02:23.038188 5346 raft_consensus.cc:359] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f9c189dbbe46de8279b2fe462ccadd" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33897 } }
I20251214 03:02:23.038396 5346 raft_consensus.cc:385] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:23.038435 5346 raft_consensus.cc:740] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f9c189dbbe46de8279b2fe462ccadd, State: Initialized, Role: FOLLOWER
I20251214 03:02:23.038884 5346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f9c189dbbe46de8279b2fe462ccadd" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33897 } }
I20251214 03:02:23.039007 5346 raft_consensus.cc:399] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:23.039067 5346 raft_consensus.cc:493] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:23.039149 5346 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:23.040083 5346 raft_consensus.cc:515] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f9c189dbbe46de8279b2fe462ccadd" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33897 } }
I20251214 03:02:23.040437 5346 leader_election.cc:304] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f9c189dbbe46de8279b2fe462ccadd; no voters:
I20251214 03:02:23.040731 5346 leader_election.cc:290] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:23.040829 5351 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:23.040964 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:23.027687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.1.157.254:33897)
1214 03:02:23.028071 (+ 384us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:23.028080 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:23.028114 (+ 34us) server_negotiation.cc:408] Connection header received
1214 03:02:23.028633 (+ 519us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:23.028650 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:23.028936 (+ 286us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:23.029246 (+ 310us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:23.030530 (+ 1284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.031401 (+ 871us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:23.032289 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.032618 (+ 329us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:23.034931 (+ 2313us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:23.034953 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:23.034965 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:23.035000 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:23.037999 (+ 2999us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:23.038501 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:23.038507 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:23.038514 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:23.038604 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:23.038829 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:23.038832 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:23.038834 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:23.039280 (+ 446us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:23.039486 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:23.039767 (+ 281us) server_negotiation.cc:300] Negotiation successful
1214 03:02:23.040038 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20251214 03:02:23.040982 5351 raft_consensus.cc:697] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [term 1 LEADER]: Becoming Leader. State: Replica: 05f9c189dbbe46de8279b2fe462ccadd, State: Running, Role: LEADER
I20251214 03:02:23.041574 5351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f9c189dbbe46de8279b2fe462ccadd" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33897 } }
I20251214 03:02:23.041941 5346 sys_catalog.cc:565] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:23.043170 5352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05f9c189dbbe46de8279b2fe462ccadd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f9c189dbbe46de8279b2fe462ccadd" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33897 } } }
I20251214 03:02:23.043324 5352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:23.043639 5353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05f9c189dbbe46de8279b2fe462ccadd. Latest consensus state: current_term: 1 leader_uuid: "05f9c189dbbe46de8279b2fe462ccadd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f9c189dbbe46de8279b2fe462ccadd" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33897 } } }
I20251214 03:02:23.043720 5353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:23.044108 5360 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:23.047513 5360 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:23.054795 5360 catalog_manager.cc:1357] Generated new cluster ID: 3ba7bc8723fe47f6a270e8130383c462
I20251214 03:02:23.054895 5360 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:23.075695 5360 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:23.076694 5360 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:23.093039 5360 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 05f9c189dbbe46de8279b2fe462ccadd: Generated new TSK 0
I20251214 03:02:23.093812 5360 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:23.152478 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:33897
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:23.270500 5374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:23.270743 5374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:23.270802 5374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:23.274071 5374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:23.274128 5374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:23.274209 5374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:23.278306 5374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:33897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:23.279255 5374 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:23.280025 5374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:23.286219 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:23.286196 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:23.286185 5382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:23.286407 5374 server_base.cc:1047] running on GCE node
I20251214 03:02:23.286770 5374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:23.287261 5374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:23.288408 5374 hybrid_clock.cc:648] HybridClock initialized: now 1765681343288390 us; error 36 us; skew 500 ppm
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:23.290771 5374 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:02:23.291740 5374 webserver.cc:492] Webserver started at http://127.1.157.193:34173/ using document root <none> and password file <none>
I20251214 03:02:23.292225 5374 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:23.292268 5374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:23.292412 5374 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:23.294031 5374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "99170da8ddef44d88e6c769b633d0491"
format_stamp: "Formatted at 2025-12-14 03:02:23 on dist-test-slave-k7z7"
server_key: "8070332f8b0dc5ffe3578e35d4ea8a14"
server_key_iv: "268be0f76c75cb20176d40651d64777b"
server_key_version: "encryptionkey@0"
I20251214 03:02:23.294451 5374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "99170da8ddef44d88e6c769b633d0491"
format_stamp: "Formatted at 2025-12-14 03:02:23 on dist-test-slave-k7z7"
server_key: "8070332f8b0dc5ffe3578e35d4ea8a14"
server_key_iv: "268be0f76c75cb20176d40651d64777b"
server_key_version: "encryptionkey@0"
I20251214 03:02:23.297664 5374 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:02:23.299810 5389 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:23.300789 5374 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:23.300885 5374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "99170da8ddef44d88e6c769b633d0491"
format_stamp: "Formatted at 2025-12-14 03:02:23 on dist-test-slave-k7z7"
server_key: "8070332f8b0dc5ffe3578e35d4ea8a14"
server_key_iv: "268be0f76c75cb20176d40651d64777b"
server_key_version: "encryptionkey@0"
I20251214 03:02:23.300980 5374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:23.315140 5374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:23.318305 5374 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:23.318490 5374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:23.319046 5374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:23.319944 5374 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:23.319991 5374 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:23.320031 5374 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:23.320060 5374 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:23.329304 5374 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:39287
I20251214 03:02:23.329352 5502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:39287 every 8 connection(s)
I20251214 03:02:23.330291 5374 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:23.339044 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 5374
I20251214 03:02:23.339148 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:02:23.339376 1655 external_mini_cluster.cc:1468] Setting key aa5a1905a127efd5c97da41ffec0a03e
I20251214 03:02:23.342957 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:23.332218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:44071 (local address 127.1.157.254:33897)
1214 03:02:23.332376 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:23.332380 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:23.333097 (+ 717us) server_negotiation.cc:408] Connection header received
1214 03:02:23.334037 (+ 940us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:23.334041 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:23.334099 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:23.334181 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:23.335935 (+ 1754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.336450 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:23.337099 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.337313 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:23.339597 (+ 2284us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:23.339620 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:23.339622 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:23.339649 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:23.341020 (+ 1371us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:23.341542 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:23.341545 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:23.341546 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:23.341592 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:23.341880 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:23.341882 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:23.341884 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:23.342039 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:23.342124 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:23.342657 (+ 533us) server_negotiation.cc:300] Negotiation successful
1214 03:02:23.342797 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251214 03:02:23.343623 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:23.332463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33897 (local address 127.1.157.193:44071)
1214 03:02:23.332936 (+ 473us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:23.332974 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:23.333819 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:23.334361 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:23.334371 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:23.334791 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:23.335761 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:23.335778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.336574 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:23.336577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:23.336979 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:23.336987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.337246 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:23.337885 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:23.337913 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:23.339416 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:23.341139 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:23.341146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:23.341159 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:23.341441 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:23.341671 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:23.341674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:23.341675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:23.341784 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:23.342138 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:23.342146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:23.342393 (+ 247us) client_negotiation.cc:769] Sending connection context
1214 03:02:23.342587 (+ 194us) client_negotiation.cc:240] Negotiation successful
1214 03:02:23.342867 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":108,"threads_started":1}
I20251214 03:02:23.344825 5503 heartbeater.cc:344] Connected to a master server at 127.1.157.254:33897
I20251214 03:02:23.345122 5503 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:23.345862 5503 heartbeater.cc:507] Master 127.1.157.254:33897 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:02:23.347663 5310 ts_manager.cc:194] Registered new tserver with Master: 99170da8ddef44d88e6c769b633d0491 (127.1.157.193:39287)
I20251214 03:02:23.349077 5310 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:44071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:23.383889 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:33897
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:23.497547 5510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:23.497779 5510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:23.497838 5510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:23.501080 5510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:23.501145 5510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:23.501268 5510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:23.505525 5510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:33897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:23.506552 5510 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:23.507332 5510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:23.513633 5515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:23.513643 5518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:23.513702 5516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:23.513899 5510 server_base.cc:1047] running on GCE node
I20251214 03:02:23.514281 5510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:23.514809 5510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:23.515964 5510 hybrid_clock.cc:648] HybridClock initialized: now 1765681343515947 us; error 29 us; skew 500 ppm
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:23.518575 5510 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:02:23.519572 5510 webserver.cc:492] Webserver started at http://127.1.157.194:38185/ using document root <none> and password file <none>
I20251214 03:02:23.520094 5510 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:23.520143 5510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:23.520347 5510 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:23.522040 5510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "847db69998194387aea3453d2545ddf4"
format_stamp: "Formatted at 2025-12-14 03:02:23 on dist-test-slave-k7z7"
server_key: "98ef833f26d4a5b27b1bbc0848d223c2"
server_key_iv: "9808aa9daeeea493ac6953d1c6e10f1d"
server_key_version: "encryptionkey@0"
I20251214 03:02:23.522491 5510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "847db69998194387aea3453d2545ddf4"
format_stamp: "Formatted at 2025-12-14 03:02:23 on dist-test-slave-k7z7"
server_key: "98ef833f26d4a5b27b1bbc0848d223c2"
server_key_iv: "9808aa9daeeea493ac6953d1c6e10f1d"
server_key_version: "encryptionkey@0"
I20251214 03:02:23.525767 5510 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:02:23.527915 5525 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:23.528925 5510 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:23.529028 5510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "847db69998194387aea3453d2545ddf4"
format_stamp: "Formatted at 2025-12-14 03:02:23 on dist-test-slave-k7z7"
server_key: "98ef833f26d4a5b27b1bbc0848d223c2"
server_key_iv: "9808aa9daeeea493ac6953d1c6e10f1d"
server_key_version: "encryptionkey@0"
I20251214 03:02:23.529116 5510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:23.560667 5510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:23.563606 5510 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:23.563782 5510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:23.564333 5510 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:23.565223 5510 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:23.565270 5510 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:23.565321 5510 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:23.565348 5510 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:23.574287 5510 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:46515
I20251214 03:02:23.574373 5638 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:46515 every 8 connection(s)
I20251214 03:02:23.575171 5510 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:02:23.579367 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 5510
I20251214 03:02:23.579458 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:02:23.579663 1655 external_mini_cluster.cc:1468] Setting key b2c5a9150cfe8f985131962262f809e8
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:23.587345 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:23.576843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:55245 (local address 127.1.157.254:33897)
1214 03:02:23.576972 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:23.576976 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:23.577571 (+ 595us) server_negotiation.cc:408] Connection header received
1214 03:02:23.578369 (+ 798us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:23.578373 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:23.578454 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:23.578523 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:23.580218 (+ 1695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.580849 (+ 631us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:23.581505 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.581682 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:23.583947 (+ 2265us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:23.583967 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:23.583970 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:23.583999 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:23.585352 (+ 1353us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:23.585894 (+ 542us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:23.585897 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:23.585899 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:23.585942 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:23.586252 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:23.586254 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:23.586256 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:23.586402 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:23.586484 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:23.587106 (+ 622us) server_negotiation.cc:300] Negotiation successful
1214 03:02:23.587224 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
I20251214 03:02:23.588299 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:23.577020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33897 (local address 127.1.157.194:55245)
1214 03:02:23.577443 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:23.577475 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:23.578180 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:23.578684 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:23.578692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:23.579055 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:23.580074 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:23.580089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.580959 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:23.580962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:23.581395 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:23.581404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.581605 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:23.582249 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:23.582273 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:23.583787 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:23.585458 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:23.585464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:23.585477 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:23.585785 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:23.586019 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:23.586022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:23.586025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:23.586153 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:23.586506 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:23.586513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:23.586814 (+ 301us) client_negotiation.cc:769] Sending connection context
1214 03:02:23.587053 (+ 239us) client_negotiation.cc:240] Negotiation successful
1214 03:02:23.587354 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
I20251214 03:02:23.589453 5639 heartbeater.cc:344] Connected to a master server at 127.1.157.254:33897
I20251214 03:02:23.589712 5639 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:23.590270 5639 heartbeater.cc:507] Master 127.1.157.254:33897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
I20251214 03:02:23.591370 5310 ts_manager.cc:194] Registered new tserver with Master: 847db69998194387aea3453d2545ddf4 (127.1.157.194:46515)
I20251214 03:02:23.591935 5310 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:55245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:23.631688 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:33897
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:23.741691 5646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:23.741935 5646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:23.741993 5646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:23.745404 5646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:23.745467 5646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:23.745548 5646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:23.749789 5646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:33897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:23.750803 5646 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:23.751605 5646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:23.758323 5651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:23.758313 5654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:23.758401 5646 server_base.cc:1047] running on GCE node
W20251214 03:02:23.758313 5652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:23.758958 5646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:23.759496 5646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:23.760659 5646 hybrid_clock.cc:648] HybridClock initialized: now 1765681343760637 us; error 40 us; skew 500 ppm
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:23.763335 5646 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:02:23.764408 5646 webserver.cc:492] Webserver started at http://127.1.157.195:37241/ using document root <none> and password file <none>
I20251214 03:02:23.764930 5646 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:23.764981 5646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:23.765149 5646 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:23.766836 5646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "81c6b01c850d481791de068294af9319"
format_stamp: "Formatted at 2025-12-14 03:02:23 on dist-test-slave-k7z7"
server_key: "45fc01f9f3acd912d9f3cb61b91d58ac"
server_key_iv: "2852bbbf573430c1d08795da690366e0"
server_key_version: "encryptionkey@0"
I20251214 03:02:23.767293 5646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "81c6b01c850d481791de068294af9319"
format_stamp: "Formatted at 2025-12-14 03:02:23 on dist-test-slave-k7z7"
server_key: "45fc01f9f3acd912d9f3cb61b91d58ac"
server_key_iv: "2852bbbf573430c1d08795da690366e0"
server_key_version: "encryptionkey@0"
I20251214 03:02:23.770851 5646 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251214 03:02:23.773015 5661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:23.774024 5646 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:23.774125 5646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "81c6b01c850d481791de068294af9319"
format_stamp: "Formatted at 2025-12-14 03:02:23 on dist-test-slave-k7z7"
server_key: "45fc01f9f3acd912d9f3cb61b91d58ac"
server_key_iv: "2852bbbf573430c1d08795da690366e0"
server_key_version: "encryptionkey@0"
I20251214 03:02:23.774235 5646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:23.787293 5646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:23.789898 5646 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:23.790068 5646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:23.790581 5646 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:23.791453 5646 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:23.791500 5646 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:23.791551 5646 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:23.791579 5646 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:23.800104 5646 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:45843
I20251214 03:02:23.800177 5774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:45843 every 8 connection(s)
I20251214 03:02:23.801072 5646 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:02:23.807679 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 5646
I20251214 03:02:23.807825 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:02:23.808125 1655 external_mini_cluster.cc:1468] Setting key 6fd62bd3d986f338f3d9e14b93377286
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:23.816267 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:23.802806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:50233 (local address 127.1.157.254:33897)
1214 03:02:23.802973 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:23.802977 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:23.803639 (+ 662us) server_negotiation.cc:408] Connection header received
1214 03:02:23.804489 (+ 850us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:23.804492 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:23.804554 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:23.804632 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:23.806454 (+ 1822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.806977 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:23.807789 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.807978 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:23.811556 (+ 3578us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:23.811574 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:23.811577 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:23.811613 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:23.813703 (+ 2090us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:23.814650 (+ 947us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:23.814654 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:23.814656 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:23.814714 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:23.815115 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:23.815119 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:23.815121 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:23.815308 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:23.815395 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:23.815995 (+ 600us) server_negotiation.cc:300] Negotiation successful
1214 03:02:23.816124 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251214 03:02:23.816956 5777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:23.803052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33897 (local address 127.1.157.195:50233)
1214 03:02:23.803496 (+ 444us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:23.803531 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:23.804273 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:23.804818 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:23.804829 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:23.805317 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:23.806312 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:23.806328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.807112 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:23.807116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:23.807623 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:23.807635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.807897 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:23.808686 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:23.808724 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:23.811303 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:23.813905 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:23.813913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:23.813928 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:23.814502 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:23.814843 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:23.814847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:23.814848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:23.814987 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:23.815438 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:23.815445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:23.815718 (+ 273us) client_negotiation.cc:769] Sending connection context
1214 03:02:23.815920 (+ 202us) client_negotiation.cc:240] Negotiation successful
1214 03:02:23.816172 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20251214 03:02:23.818337 5775 heartbeater.cc:344] Connected to a master server at 127.1.157.254:33897
I20251214 03:02:23.818634 5775 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:23.819250 5775 heartbeater.cc:507] Master 127.1.157.254:33897 requested a full tablet report, sending...
I20251214 03:02:23.820434 5310 ts_manager.cc:194] Registered new tserver with Master: 81c6b01c850d481791de068294af9319 (127.1.157.195:45843)
I20251214 03:02:23.821025 5310 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:50233
I20251214 03:02:23.823002 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681343, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681343, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:23.846220 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:23.838833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.1.157.254:33897)
1214 03:02:23.838972 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:23.838975 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:23.839026 (+ 51us) server_negotiation.cc:408] Connection header received
1214 03:02:23.839141 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:23.839143 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:23.839190 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:23.839278 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:23.840063 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.840559 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:23.841115 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:23.841277 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:23.843395 (+ 2118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:23.843414 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:23.843416 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:23.843442 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:23.844832 (+ 1390us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:23.845218 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:23.845222 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:23.845224 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:23.845267 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:23.845533 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:23.845536 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:23.845537 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:23.845712 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:23.845806 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:23.846001 (+ 195us) server_negotiation.cc:300] Negotiation successful
1214 03:02:23.846097 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20251214 03:02:23.846689 5310 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54800
I20251214 03:02:23.847474 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 5374
I20251214 03:02:23.853542 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 5510
I20251214 03:02:23.858358 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 5646
I20251214 03:02:23.863138 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 5278
2025-12-14T03:02:23Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3030 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5787](info): set up 2 sockets
Dec 14 03:02:23 dist-test-slave-k7z7 krb5kdc[5787](info): commencing operation
krb5kdc: starting...
W20251214 03:02:25.713323 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.830s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:02:25 dist-test-slave-k7z7 krb5kdc[5787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:02:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:25.851792 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:39481
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:39481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:25.956926 5803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:25.957156 5803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:25.957252 5803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:25.960516 5803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:25.960577 5803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:25.960598 5803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:25.960615 5803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:25.960631 5803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:25.964937 5803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:39481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:39481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:25.966033 5803 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:25.966825 5803 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:25.972481 5811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:25.972469 5809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:25.972469 5808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:25.972602 5803 server_base.cc:1047] running on GCE node
I20251214 03:02:25.973359 5803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:25.974318 5803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:25.975466 5803 hybrid_clock.cc:648] HybridClock initialized: now 1765681345975455 us; error 34 us; skew 500 ppm
Dec 14 03:02:25 dist-test-slave-k7z7 krb5kdc[5787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681345, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:25.978003 5803 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:02:25.979079 5803 webserver.cc:492] Webserver started at http://127.1.157.254:43795/ using document root <none> and password file <none>
I20251214 03:02:25.979612 5803 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:25.979662 5803 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:25.979836 5803 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:25.981570 5803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "f96088ff823f4189b8e01d3f8eeaa52c"
format_stamp: "Formatted at 2025-12-14 03:02:25 on dist-test-slave-k7z7"
server_key: "31fd02147e9ac5e1e4bbcf06567b4049"
server_key_iv: "751fca5493ddfaad71008406ecb0c082"
server_key_version: "encryptionkey@0"
I20251214 03:02:25.982028 5803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "f96088ff823f4189b8e01d3f8eeaa52c"
format_stamp: "Formatted at 2025-12-14 03:02:25 on dist-test-slave-k7z7"
server_key: "31fd02147e9ac5e1e4bbcf06567b4049"
server_key_iv: "751fca5493ddfaad71008406ecb0c082"
server_key_version: "encryptionkey@0"
I20251214 03:02:25.985394 5803 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:02:25.987684 5818 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:25.988692 5803 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:25.988808 5803 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "f96088ff823f4189b8e01d3f8eeaa52c"
format_stamp: "Formatted at 2025-12-14 03:02:25 on dist-test-slave-k7z7"
server_key: "31fd02147e9ac5e1e4bbcf06567b4049"
server_key_iv: "751fca5493ddfaad71008406ecb0c082"
server_key_version: "encryptionkey@0"
I20251214 03:02:25.988905 5803 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:26.009009 5803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:26.012683 5803 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:26.012893 5803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:26.021405 5803 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:39481
I20251214 03:02:26.021401 5870 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:39481 every 8 connection(s)
I20251214 03:02:26.022457 5803 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:26.025259 5871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:26.027510 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 5803
I20251214 03:02:26.027650 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:26.027886 1655 external_mini_cluster.cc:1468] Setting key 1bd7283e54b0efcbce91e52c7c516a63
I20251214 03:02:26.031250 5871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c: Bootstrap starting.
I20251214 03:02:26.033327 5871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:26.033990 5871 log.cc:826] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c: Log is configured to *not* fsync() on all Append() calls
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[5787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:26.036268 5871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c: No bootstrap required, opened a new log
I20251214 03:02:26.038712 5871 raft_consensus.cc:359] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96088ff823f4189b8e01d3f8eeaa52c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39481 } }
I20251214 03:02:26.038888 5871 raft_consensus.cc:385] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:26.038949 5871 raft_consensus.cc:740] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96088ff823f4189b8e01d3f8eeaa52c, State: Initialized, Role: FOLLOWER
I20251214 03:02:26.039366 5871 consensus_queue.cc:260] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96088ff823f4189b8e01d3f8eeaa52c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39481 } }
I20251214 03:02:26.039497 5871 raft_consensus.cc:399] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:26.039572 5871 raft_consensus.cc:493] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:26.039661 5871 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:26.040656 5871 raft_consensus.cc:515] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96088ff823f4189b8e01d3f8eeaa52c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39481 } }
I20251214 03:02:26.040885 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:26.029250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.1.157.254:39481)
1214 03:02:26.029676 (+ 426us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:26.029687 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:26.029721 (+ 34us) server_negotiation.cc:408] Connection header received
1214 03:02:26.030348 (+ 627us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:26.030367 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:26.030689 (+ 322us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:26.031020 (+ 331us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:26.032460 (+ 1440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.033266 (+ 806us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:26.033933 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.034217 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:26.036651 (+ 2434us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:26.036670 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:26.036679 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:26.036710 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:26.038387 (+ 1677us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.038752 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.038757 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.038761 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.038827 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.039096 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.039099 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.039101 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.039407 (+ 306us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:26.039593 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:26.039885 (+ 292us) server_negotiation.cc:300] Negotiation successful
1214 03:02:26.040135 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":127,"threads_started":1}
I20251214 03:02:26.041016 5871 leader_election.cc:304] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f96088ff823f4189b8e01d3f8eeaa52c; no voters:
I20251214 03:02:26.041360 5871 leader_election.cc:290] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:26.041543 5876 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:26.041766 5876 raft_consensus.cc:697] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [term 1 LEADER]: Becoming Leader. State: Replica: f96088ff823f4189b8e01d3f8eeaa52c, State: Running, Role: LEADER
I20251214 03:02:26.042068 5876 consensus_queue.cc:237] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96088ff823f4189b8e01d3f8eeaa52c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39481 } }
I20251214 03:02:26.042443 5871 sys_catalog.cc:565] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:26.043244 5876 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f96088ff823f4189b8e01d3f8eeaa52c. Latest consensus state: current_term: 1 leader_uuid: "f96088ff823f4189b8e01d3f8eeaa52c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96088ff823f4189b8e01d3f8eeaa52c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39481 } } }
I20251214 03:02:26.043888 5876 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:26.043787 5877 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f96088ff823f4189b8e01d3f8eeaa52c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96088ff823f4189b8e01d3f8eeaa52c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39481 } } }
I20251214 03:02:26.044149 5877 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:26.044250 5885 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:26.047441 5885 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:26.052733 5885 catalog_manager.cc:1357] Generated new cluster ID: 6609641f5dfc40df84359b9e407394a6
I20251214 03:02:26.052794 5885 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:26.061482 5885 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:26.062350 5885 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:26.068982 5885 catalog_manager.cc:6027] T 00000000000000000000000000000000 P f96088ff823f4189b8e01d3f8eeaa52c: Generated new TSK 0
I20251214 03:02:26.069685 5885 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:26.130640 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:39481
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:26.249444 5899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:26.249691 5899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:26.249751 5899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:26.253293 5899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:26.253382 5899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:26.253468 5899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:26.258497 5899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:39481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:26.259676 5899 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:26.260537 5899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:26.267535 5905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:26.267546 5904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:26.267693 5899 server_base.cc:1047] running on GCE node
W20251214 03:02:26.267547 5907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:26.268218 5899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:26.268778 5899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:26.269923 5899 hybrid_clock.cc:648] HybridClock initialized: now 1765681346269906 us; error 31 us; skew 500 ppm
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[5787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:26.272388 5899 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:02:26.273474 5899 webserver.cc:492] Webserver started at http://127.1.157.193:43193/ using document root <none> and password file <none>
I20251214 03:02:26.274004 5899 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:26.274055 5899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:26.274227 5899 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:26.275928 5899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "e68ceec49fdc46afa04dd8a3b1a7b4d8"
format_stamp: "Formatted at 2025-12-14 03:02:26 on dist-test-slave-k7z7"
server_key: "b7635f3eded87bd89e643099a1b84b53"
server_key_iv: "55f3bf817df561b8ace084c7900894dc"
server_key_version: "encryptionkey@0"
I20251214 03:02:26.276422 5899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "e68ceec49fdc46afa04dd8a3b1a7b4d8"
format_stamp: "Formatted at 2025-12-14 03:02:26 on dist-test-slave-k7z7"
server_key: "b7635f3eded87bd89e643099a1b84b53"
server_key_iv: "55f3bf817df561b8ace084c7900894dc"
server_key_version: "encryptionkey@0"
I20251214 03:02:26.279764 5899 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251214 03:02:26.281929 5914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:26.282895 5899 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:26.282996 5899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "e68ceec49fdc46afa04dd8a3b1a7b4d8"
format_stamp: "Formatted at 2025-12-14 03:02:26 on dist-test-slave-k7z7"
server_key: "b7635f3eded87bd89e643099a1b84b53"
server_key_iv: "55f3bf817df561b8ace084c7900894dc"
server_key_version: "encryptionkey@0"
I20251214 03:02:26.283080 5899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:26.304385 5899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:26.307269 5899 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:26.307442 5899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:26.307986 5899 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:26.308851 5899 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:26.308899 5899 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:26.308951 5899 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:26.308976 5899 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:26.317673 5899 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:35115
I20251214 03:02:26.317705 6027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:35115 every 8 connection(s)
I20251214 03:02:26.318614 5899 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:02:26.327227 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 5899
I20251214 03:02:26.327337 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[5787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:26.327605 1655 external_mini_cluster.cc:1468] Setting key 9d497514f4f251f2b44e1ab38b926179
I20251214 03:02:26.333509 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:26.320597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:42837 (local address 127.1.157.254:39481)
1214 03:02:26.320787 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:26.320791 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:26.321599 (+ 808us) server_negotiation.cc:408] Connection header received
1214 03:02:26.322474 (+ 875us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:26.322477 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:26.322531 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:26.322622 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:26.324465 (+ 1843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.324992 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:26.325763 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.325923 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:26.328180 (+ 2257us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:26.328204 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:26.328207 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:26.328238 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:26.331214 (+ 2976us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.331813 (+ 599us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.331817 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.331819 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.331879 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.332200 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.332203 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.332205 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.332420 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:26.332537 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:26.333126 (+ 589us) server_negotiation.cc:300] Negotiation successful
1214 03:02:26.333313 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251214 03:02:26.334077 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:26.320867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:39481 (local address 127.1.157.193:42837)
1214 03:02:26.321459 (+ 592us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:26.321497 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:26.322248 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:26.322853 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:26.322863 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:26.323295 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:26.324263 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:26.324281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.325135 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:26.325139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:26.325617 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:26.325627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.325834 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:26.326457 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:26.326486 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:26.328002 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:26.331395 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:26.331401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:26.331414 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:26.331706 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:26.331990 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:26.331993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:26.331995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:26.332110 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:26.332574 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:26.332580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:26.332855 (+ 275us) client_negotiation.cc:769] Sending connection context
1214 03:02:26.333049 (+ 194us) client_negotiation.cc:240] Negotiation successful
1214 03:02:26.333338 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":131,"threads_started":1}
I20251214 03:02:26.335263 6028 heartbeater.cc:344] Connected to a master server at 127.1.157.254:39481
I20251214 03:02:26.335528 6028 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:26.336079 6028 heartbeater.cc:507] Master 127.1.157.254:39481 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:02:26.337741 5835 ts_manager.cc:194] Registered new tserver with Master: e68ceec49fdc46afa04dd8a3b1a7b4d8 (127.1.157.193:35115)
I20251214 03:02:26.339077 5835 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:42837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:26.383658 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:39481
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:26.488744 6035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:26.488971 6035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:26.489050 6035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:26.492352 6035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:26.492425 6035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:26.492533 6035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:26.496762 6035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:39481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:26.497805 6035 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:26.498607 6035 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:26.504861 6040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:26.504868 6041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:26.504868 6043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:26.504973 6035 server_base.cc:1047] running on GCE node
I20251214 03:02:26.505450 6035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:26.505970 6035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:26.507138 6035 hybrid_clock.cc:648] HybridClock initialized: now 1765681346507125 us; error 28 us; skew 500 ppm
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[5787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:26.509694 6035 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:02:26.510699 6035 webserver.cc:492] Webserver started at http://127.1.157.194:41611/ using document root <none> and password file <none>
I20251214 03:02:26.511224 6035 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:26.511271 6035 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:26.511435 6035 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:26.513023 6035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "ed66a2f7536a4027a680917c45d0f32d"
format_stamp: "Formatted at 2025-12-14 03:02:26 on dist-test-slave-k7z7"
server_key: "18ebeff8e06e2eb6b62a83b02f119cbd"
server_key_iv: "826c7543ad234e376c762ce31450a799"
server_key_version: "encryptionkey@0"
I20251214 03:02:26.513502 6035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "ed66a2f7536a4027a680917c45d0f32d"
format_stamp: "Formatted at 2025-12-14 03:02:26 on dist-test-slave-k7z7"
server_key: "18ebeff8e06e2eb6b62a83b02f119cbd"
server_key_iv: "826c7543ad234e376c762ce31450a799"
server_key_version: "encryptionkey@0"
I20251214 03:02:26.516605 6035 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251214 03:02:26.518687 6050 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:26.519642 6035 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:26.519738 6035 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "ed66a2f7536a4027a680917c45d0f32d"
format_stamp: "Formatted at 2025-12-14 03:02:26 on dist-test-slave-k7z7"
server_key: "18ebeff8e06e2eb6b62a83b02f119cbd"
server_key_iv: "826c7543ad234e376c762ce31450a799"
server_key_version: "encryptionkey@0"
I20251214 03:02:26.519819 6035 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:26.536370 6035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:26.539245 6035 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:26.539412 6035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:26.539942 6035 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:26.540819 6035 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:26.540867 6035 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:26.540920 6035 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:26.540946 6035 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:26.549940 6035 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:44865
I20251214 03:02:26.549964 6163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:44865 every 8 connection(s)
I20251214 03:02:26.550915 6035 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:02:26.559180 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 6035
I20251214 03:02:26.559275 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:02:26.559499 1655 external_mini_cluster.cc:1468] Setting key 32c1c5d2ca44049c9c00a99a053bb697
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[5787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:26.563627 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:26.552950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:59745 (local address 127.1.157.254:39481)
1214 03:02:26.553123 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:26.553127 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:26.553847 (+ 720us) server_negotiation.cc:408] Connection header received
1214 03:02:26.554680 (+ 833us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:26.554683 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:26.554743 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:26.554816 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:26.556480 (+ 1664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.556977 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:26.557767 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.557959 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:26.560264 (+ 2305us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:26.560284 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:26.560286 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:26.560313 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:26.561760 (+ 1447us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.562244 (+ 484us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.562247 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.562249 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.562294 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.562561 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.562565 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.562567 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.562727 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:26.562811 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:26.563328 (+ 517us) server_negotiation.cc:300] Negotiation successful
1214 03:02:26.563458 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251214 03:02:26.564257 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:26.553256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:39481 (local address 127.1.157.194:59745)
1214 03:02:26.553708 (+ 452us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:26.553744 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:26.554490 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:26.554980 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:26.554988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:26.555406 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:26.556329 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:26.556343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.557089 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:26.557092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:26.557616 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:26.557627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.557875 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:26.558502 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:26.558529 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:26.560085 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:26.561882 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:26.561889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:26.561903 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:26.562143 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:26.562373 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:26.562376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:26.562378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:26.562480 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:26.562824 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:26.562832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:26.563073 (+ 241us) client_negotiation.cc:769] Sending connection context
1214 03:02:26.563262 (+ 189us) client_negotiation.cc:240] Negotiation successful
1214 03:02:26.563555 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20251214 03:02:26.565681 6164 heartbeater.cc:344] Connected to a master server at 127.1.157.254:39481
I20251214 03:02:26.565965 6164 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:26.566533 6164 heartbeater.cc:507] Master 127.1.157.254:39481 requested a full tablet report, sending...
I20251214 03:02:26.567902 5835 ts_manager.cc:194] Registered new tserver with Master: ed66a2f7536a4027a680917c45d0f32d (127.1.157.194:44865)
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
I20251214 03:02:26.568730 5835 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:59745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:26.608527 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:39481
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:26.714035 6171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:26.714282 6171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:26.714361 6171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:26.717972 6171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:26.718070 6171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:26.718184 6171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:26.722756 6171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:39481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:26.723902 6171 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:26.724782 6171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:26.731557 6176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:26.731564 6177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:26.731564 6179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:26.732004 6171 server_base.cc:1047] running on GCE node
I20251214 03:02:26.732394 6171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:26.732940 6171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:26.734128 6171 hybrid_clock.cc:648] HybridClock initialized: now 1765681346734106 us; error 36 us; skew 500 ppm
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[5787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:26.737030 6171 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:02:26.738243 6171 webserver.cc:492] Webserver started at http://127.1.157.195:41717/ using document root <none> and password file <none>
I20251214 03:02:26.738794 6171 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:26.738852 6171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:26.739032 6171 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:26.740785 6171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "8f00320546e547069246c21dfba4c2fe"
format_stamp: "Formatted at 2025-12-14 03:02:26 on dist-test-slave-k7z7"
server_key: "555663b15b435132148b39d4d7863757"
server_key_iv: "87f4abab83a19619287d5cb4f8befc07"
server_key_version: "encryptionkey@0"
I20251214 03:02:26.741310 6171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "8f00320546e547069246c21dfba4c2fe"
format_stamp: "Formatted at 2025-12-14 03:02:26 on dist-test-slave-k7z7"
server_key: "555663b15b435132148b39d4d7863757"
server_key_iv: "87f4abab83a19619287d5cb4f8befc07"
server_key_version: "encryptionkey@0"
I20251214 03:02:26.744980 6171 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:02:26.747205 6186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:26.748188 6171 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:26.748291 6171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "8f00320546e547069246c21dfba4c2fe"
format_stamp: "Formatted at 2025-12-14 03:02:26 on dist-test-slave-k7z7"
server_key: "555663b15b435132148b39d4d7863757"
server_key_iv: "87f4abab83a19619287d5cb4f8befc07"
server_key_version: "encryptionkey@0"
I20251214 03:02:26.748376 6171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:26.795337 6171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:26.798661 6171 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:26.798874 6171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:26.799494 6171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:26.800434 6171 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:26.800489 6171 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:26.800547 6171 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:26.800575 6171 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:26.810103 6171 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:41345
I20251214 03:02:26.810174 6299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:41345 every 8 connection(s)
I20251214 03:02:26.811058 6171 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:02:26.814467 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 6171
I20251214 03:02:26.814590 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:02:26.814844 1655 external_mini_cluster.cc:1468] Setting key 7f7c499b71697b183ea113fefdac1d7d
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[5787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:26.824092 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:26.812808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:55233 (local address 127.1.157.254:39481)
1214 03:02:26.812941 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:26.812945 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:26.813682 (+ 737us) server_negotiation.cc:408] Connection header received
1214 03:02:26.814510 (+ 828us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:26.814514 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:26.814572 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:26.814649 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:26.816823 (+ 2174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.817417 (+ 594us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:26.818071 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.818208 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:26.820532 (+ 2324us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:26.820549 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:26.820552 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:26.820579 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:26.822104 (+ 1525us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.822629 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.822633 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.822634 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.822688 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.823034 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.823037 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.823038 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.823209 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:26.823318 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:26.823835 (+ 517us) server_negotiation.cc:300] Negotiation successful
1214 03:02:26.823959 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251214 03:02:26.824707 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:26.813057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:39481 (local address 127.1.157.195:55233)
1214 03:02:26.813547 (+ 490us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:26.813583 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:26.814308 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:26.814865 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:26.814877 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:26.815375 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:26.816647 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:26.816676 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.817544 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:26.817547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:26.817964 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:26.817972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.818191 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:26.818811 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:26.818840 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:26.820384 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:26.822247 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:26.822253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:26.822264 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:26.822524 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:26.822813 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:26.822816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:26.822817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:26.822920 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:26.823319 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:26.823324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:26.823611 (+ 287us) client_negotiation.cc:769] Sending connection context
1214 03:02:26.823789 (+ 178us) client_negotiation.cc:240] Negotiation successful
1214 03:02:26.824023 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":105,"threads_started":1}
I20251214 03:02:26.825693 6300 heartbeater.cc:344] Connected to a master server at 127.1.157.254:39481
I20251214 03:02:26.825919 6300 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:26.826414 6300 heartbeater.cc:507] Master 127.1.157.254:39481 requested a full tablet report, sending...
I20251214 03:02:26.827549 5835 ts_manager.cc:194] Registered new tserver with Master: 8f00320546e547069246c21dfba4c2fe (127.1.157.195:41345)
I20251214 03:02:26.828241 5835 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:55233
I20251214 03:02:26.828567 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[5787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[5787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.193@KRBTEST.COM
I20251214 03:02:26.854501 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:26.843334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.1.157.193:35115)
1214 03:02:26.843849 (+ 515us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:26.843857 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:26.843889 (+ 32us) server_negotiation.cc:408] Connection header received
1214 03:02:26.844004 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:26.844011 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:26.844268 (+ 257us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:26.844444 (+ 176us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:26.845810 (+ 1366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.846855 (+ 1045us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:26.847863 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.848137 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:26.850610 (+ 2473us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:26.850699 (+ 89us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:26.850715 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:26.850759 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:26.852567 (+ 1808us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.853040 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.853046 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.853047 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.853107 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.853531 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.853534 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.853535 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.853750 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:26.853922 (+ 172us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:26.854132 (+ 210us) server_negotiation.cc:300] Negotiation successful
1214 03:02:26.854307 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":104,"threads_started":1}
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[5787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:26.865420 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:26.857384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.1.157.254:39481)
1214 03:02:26.857566 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:26.857570 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:26.857586 (+ 16us) server_negotiation.cc:408] Connection header received
1214 03:02:26.857709 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:26.857712 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:26.857759 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:26.857825 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:26.858557 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.859050 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:26.859662 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:26.859845 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:26.862191 (+ 2346us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:26.862220 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:26.862222 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:26.862254 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:26.863878 (+ 1624us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.864307 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.864311 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.864314 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.864365 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:26.864582 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:26.864585 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:26.864587 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:26.864779 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:26.864895 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:26.865106 (+ 211us) server_negotiation.cc:300] Negotiation successful
1214 03:02:26.865257 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20251214 03:02:26.865864 5835 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37328
I20251214 03:02:26.866644 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 5899
I20251214 03:02:26.872001 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 6035
I20251214 03:02:26.876798 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 6171
I20251214 03:02:26.881372 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 5803
2025-12-14T03:02:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3018 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[6312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[6312](info): set up 2 sockets
Dec 14 03:02:26 dist-test-slave-k7z7 krb5kdc[6312](info): commencing operation
krb5kdc: starting...
W20251214 03:02:28.717626 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.816s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:02:28 dist-test-slave-k7z7 krb5kdc[6312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:02:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:28.864670 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:39061
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:39061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:28.984838 6328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:28.985051 6328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:28.985126 6328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:28.988344 6328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:28.988402 6328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:28.988435 6328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:28.988467 6328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:28.988499 6328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:28.992674 6328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:39061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:28.993698 6328 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:28.994468 6328 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:28.999519 6333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:28.999562 6334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:28.999715 6328 server_base.cc:1047] running on GCE node
W20251214 03:02:28.999564 6336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:29.000278 6328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:29.001087 6328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:29.002223 6328 hybrid_clock.cc:648] HybridClock initialized: now 1765681349002209 us; error 29 us; skew 500 ppm
Dec 14 03:02:28 dist-test-slave-k7z7 krb5kdc[6312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681348, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:29.004701 6328 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:02:29.005474 6328 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:29.005538 6328 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:29.005709 6328 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:29.007784 6328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "238ec04c775e414a9a8a715bbc189638"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "c80defd3c42cbdfdd0b55e4edb34d9d7"
server_key_iv: "34f79a8edc6822c8b447eff185041989"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.008327 6328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "238ec04c775e414a9a8a715bbc189638"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "c80defd3c42cbdfdd0b55e4edb34d9d7"
server_key_iv: "34f79a8edc6822c8b447eff185041989"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.011962 6328 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251214 03:02:29.014343 6342 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:29.015408 6328 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:29.015533 6328 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "238ec04c775e414a9a8a715bbc189638"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "c80defd3c42cbdfdd0b55e4edb34d9d7"
server_key_iv: "34f79a8edc6822c8b447eff185041989"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.015633 6328 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:29.037410 6328 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:29.040287 6328 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:29.040453 6328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:29.047408 6328 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:39061
I20251214 03:02:29.047399 6394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:39061 every 8 connection(s)
I20251214 03:02:29.048028 6328 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:29.050679 6395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:29.050837 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 6328
I20251214 03:02:29.050942 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:29.051182 1655 external_mini_cluster.cc:1468] Setting key e227c5f9ee0697d7fa9f7464f11ef3fd
I20251214 03:02:29.055961 6395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638: Bootstrap starting.
I20251214 03:02:29.058524 6395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638: Neither blocks nor log segments found. Creating new log.
Dec 14 03:02:29 dist-test-slave-k7z7 krb5kdc[6312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:29.059206 6395 log.cc:826] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:29.061228 6395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638: No bootstrap required, opened a new log
I20251214 03:02:29.063990 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.052856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.1.157.254:39061)
1214 03:02:29.053294 (+ 438us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.053304 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.053337 (+ 33us) server_negotiation.cc:408] Connection header received
1214 03:02:29.053854 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.053870 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.054156 (+ 286us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.054442 (+ 286us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:29.055303 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.056036 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.056890 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.057101 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.059425 (+ 2324us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:29.059443 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:29.059465 (+ 22us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:29.059503 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:29.061400 (+ 1897us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:29.061845 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:29.061850 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:29.061856 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:29.061944 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:29.062151 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:29.062155 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:29.062157 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:29.062568 (+ 411us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:29.062744 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.063001 (+ 257us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.063256 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20251214 03:02:29.064277 6395 raft_consensus.cc:359] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238ec04c775e414a9a8a715bbc189638" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39061 } }
I20251214 03:02:29.064481 6395 raft_consensus.cc:385] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:29.064540 6395 raft_consensus.cc:740] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238ec04c775e414a9a8a715bbc189638, State: Initialized, Role: FOLLOWER
I20251214 03:02:29.064976 6395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238ec04c775e414a9a8a715bbc189638" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39061 } }
I20251214 03:02:29.065126 6395 raft_consensus.cc:399] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:29.065213 6395 raft_consensus.cc:493] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:29.065302 6395 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:29.066169 6395 raft_consensus.cc:515] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238ec04c775e414a9a8a715bbc189638" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39061 } }
I20251214 03:02:29.066491 6395 leader_election.cc:304] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 238ec04c775e414a9a8a715bbc189638; no voters:
I20251214 03:02:29.066740 6395 leader_election.cc:290] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:29.066836 6400 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:29.067025 6400 raft_consensus.cc:697] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [term 1 LEADER]: Becoming Leader. State: Replica: 238ec04c775e414a9a8a715bbc189638, State: Running, Role: LEADER
I20251214 03:02:29.067317 6400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238ec04c775e414a9a8a715bbc189638" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39061 } }
I20251214 03:02:29.067826 6395 sys_catalog.cc:565] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:29.068313 6402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 238ec04c775e414a9a8a715bbc189638. Latest consensus state: current_term: 1 leader_uuid: "238ec04c775e414a9a8a715bbc189638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238ec04c775e414a9a8a715bbc189638" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39061 } } }
I20251214 03:02:29.068974 6402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:29.068873 6401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "238ec04c775e414a9a8a715bbc189638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238ec04c775e414a9a8a715bbc189638" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 39061 } } }
I20251214 03:02:29.069308 6401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:29.070084 6409 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:29.072460 6409 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:29.077201 6409 catalog_manager.cc:1357] Generated new cluster ID: d8b0fbe66dc34bed97cbfaca22d142a7
I20251214 03:02:29.077303 6409 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:29.113260 6409 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:29.114571 6409 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:29.126910 6409 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638: Generated new TSK 0
I20251214 03:02:29.127840 6409 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:29.183548 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:39061
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:29.294893 6423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:29.295126 6423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:29.295205 6423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:29.298574 6423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:29.298637 6423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:29.298732 6423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:29.302944 6423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:29.303961 6423 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:29.304770 6423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:29.310851 6428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:29.311089 6423 server_base.cc:1047] running on GCE node
W20251214 03:02:29.310868 6429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:29.310851 6431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:29.311573 6423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:29.312106 6423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:29.313234 6423 hybrid_clock.cc:648] HybridClock initialized: now 1765681349313219 us; error 28 us; skew 500 ppm
Dec 14 03:02:29 dist-test-slave-k7z7 krb5kdc[6312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681349, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:29.315789 6423 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:02:29.316507 6423 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:29.316571 6423 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:29.316749 6423 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:29.318579 6423 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "8bf498864fe4ccdff12375b038b17c28"
server_key_iv: "e4775eadb1578c153d5ee53f5f6e8d64"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.319051 6423 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "8bf498864fe4ccdff12375b038b17c28"
server_key_iv: "e4775eadb1578c153d5ee53f5f6e8d64"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.322623 6423 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251214 03:02:29.324827 6437 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:29.325911 6423 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:29.326023 6423 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "8bf498864fe4ccdff12375b038b17c28"
server_key_iv: "e4775eadb1578c153d5ee53f5f6e8d64"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.326117 6423 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:29.351387 6423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:29.354259 6423 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:29.354434 6423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:29.354971 6423 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:29.355856 6423 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:29.355906 6423 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:29.355957 6423 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:29.355981 6423 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:29.364596 6423 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:36123
I20251214 03:02:29.364688 6550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:36123 every 8 connection(s)
I20251214 03:02:29.365144 6423 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:02:29.369383 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 6423
I20251214 03:02:29.369516 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:02:29.369745 1655 external_mini_cluster.cc:1468] Setting key a1deb2ac65cee6f5db095f9a129b5602
Dec 14 03:02:29 dist-test-slave-k7z7 krb5kdc[6312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681349, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:29.377607 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.366745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:46125 (local address 127.1.157.254:39061)
1214 03:02:29.366873 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.366877 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.367512 (+ 635us) server_negotiation.cc:408] Connection header received
1214 03:02:29.368328 (+ 816us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.368332 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.368389 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.368461 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:29.370181 (+ 1720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.370825 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.371464 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.371631 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.374139 (+ 2508us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:29.374156 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:29.374159 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:29.374184 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:29.375704 (+ 1520us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:29.376188 (+ 484us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:29.376191 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:29.376193 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:29.376257 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:29.376526 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:29.376530 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:29.376532 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:29.376724 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:29.376821 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.377309 (+ 488us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.377432 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251214 03:02:29.378188 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.366972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:39061 (local address 127.1.157.193:46125)
1214 03:02:29.367380 (+ 408us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:29.367414 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:29.368138 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:29.368619 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:29.368627 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:29.368988 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:29.370043 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.370058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.370934 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.370938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:29.371357 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.371365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.371557 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.372168 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:29.372192 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:29.373969 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:29.375840 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:29.375845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:29.375855 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:29.376079 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:29.376351 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:29.376354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:29.376356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:29.376448 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:29.376815 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:29.376821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:29.377044 (+ 223us) client_negotiation.cc:769] Sending connection context
1214 03:02:29.377259 (+ 215us) client_negotiation.cc:240] Negotiation successful
1214 03:02:29.377500 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:02:29.379299 6551 heartbeater.cc:344] Connected to a master server at 127.1.157.254:39061
I20251214 03:02:29.379554 6551 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:29.380010 6551 heartbeater.cc:507] Master 127.1.157.254:39061 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
I20251214 03:02:29.381440 6359 ts_manager.cc:194] Registered new tserver with Master: e75ec5a2ebb54786a5a1a41e890f9e2f (127.1.157.193:36123)
I20251214 03:02:29.382856 6359 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:46125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:29.413571 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:39061
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:29.518874 6558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:29.519074 6558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:29.519140 6558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:29.522557 6558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:29.522639 6558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:29.522737 6558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:29.527320 6558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:29.528419 6558 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:29.529304 6558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:29.536652 6564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:29.536787 6558 server_base.cc:1047] running on GCE node
W20251214 03:02:29.536652 6566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:29.536652 6563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:29.537619 6558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:29.538260 6558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:29.539477 6558 hybrid_clock.cc:648] HybridClock initialized: now 1765681349539464 us; error 48 us; skew 500 ppm
Dec 14 03:02:29 dist-test-slave-k7z7 krb5kdc[6312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681349, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:29.542402 6558 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:02:29.543190 6558 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:29.543264 6558 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:29.543448 6558 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:29.545208 6558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "04cd16db5dab4ed1a1c858665eb81141"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "7a603e959b131759e1345b4e16b08659"
server_key_iv: "3501175c6a61d6fe5c77906a5de906fa"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.545682 6558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "04cd16db5dab4ed1a1c858665eb81141"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "7a603e959b131759e1345b4e16b08659"
server_key_iv: "3501175c6a61d6fe5c77906a5de906fa"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.548969 6558 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:02:29.552455 6572 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:29.554289 6558 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:02:29.554414 6558 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "04cd16db5dab4ed1a1c858665eb81141"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "7a603e959b131759e1345b4e16b08659"
server_key_iv: "3501175c6a61d6fe5c77906a5de906fa"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.554527 6558 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:29.577589 6558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:29.580570 6558 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:29.580747 6558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:29.581342 6558 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:29.582252 6558 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:29.582301 6558 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:29.582345 6558 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:29.582372 6558 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:29.591432 6558 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:40119
I20251214 03:02:29.591511 6685 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:40119 every 8 connection(s)
I20251214 03:02:29.591959 6558 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:02:29.599051 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 6558
I20251214 03:02:29.599159 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:02:29.599365 1655 external_mini_cluster.cc:1468] Setting key 504a14bfb1393d73cb1e71643c9aac73
Dec 14 03:02:29 dist-test-slave-k7z7 krb5kdc[6312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681349, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:29.604300 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.593632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:55221 (local address 127.1.157.254:39061)
1214 03:02:29.593789 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.593793 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.594484 (+ 691us) server_negotiation.cc:408] Connection header received
1214 03:02:29.595345 (+ 861us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.595349 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.595411 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.595483 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:29.597125 (+ 1642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.597670 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.598308 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.598479 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.600779 (+ 2300us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:29.600797 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:29.600800 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:29.600828 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:29.602337 (+ 1509us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:29.602912 (+ 575us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:29.602915 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:29.602916 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:29.602956 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:29.603248 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:29.603252 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:29.603254 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:29.603437 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:29.603547 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.604012 (+ 465us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.604132 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251214 03:02:29.604888 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.593872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:39061 (local address 127.1.157.194:55221)
1214 03:02:29.594327 (+ 455us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:29.594365 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:29.595142 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:29.595650 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:29.595658 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:29.596042 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:29.596966 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.596979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.597790 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.597793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:29.598197 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.598205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.598398 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.599015 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:29.599050 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:29.600615 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:29.602490 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:29.602496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:29.602510 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:29.602819 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:29.603030 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:29.603033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:29.603035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:29.603157 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:29.603554 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:29.603560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:29.603787 (+ 227us) client_negotiation.cc:769] Sending connection context
1214 03:02:29.603956 (+ 169us) client_negotiation.cc:240] Negotiation successful
1214 03:02:29.604200 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20251214 03:02:29.606053 6686 heartbeater.cc:344] Connected to a master server at 127.1.157.254:39061
I20251214 03:02:29.606282 6686 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:29.606761 6686 heartbeater.cc:507] Master 127.1.157.254:39061 requested a full tablet report, sending...
I20251214 03:02:29.607779 6359 ts_manager.cc:194] Registered new tserver with Master: 04cd16db5dab4ed1a1c858665eb81141 (127.1.157.194:40119)
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
I20251214 03:02:29.608332 6359 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:55221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:29.643937 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:39061
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:29.745888 6693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:29.746086 6693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:29.746148 6693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:29.749482 6693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:29.749544 6693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:29.749641 6693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:29.753895 6693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:29.754897 6693 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:29.755697 6693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:29.761806 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:29.762248 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:29.762293 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:29.762482 6693 server_base.cc:1047] running on GCE node
I20251214 03:02:29.762804 6693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:29.763310 6693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:29.764472 6693 hybrid_clock.cc:648] HybridClock initialized: now 1765681349764458 us; error 41 us; skew 500 ppm
Dec 14 03:02:29 dist-test-slave-k7z7 krb5kdc[6312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681349, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:29.766889 6693 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:02:29.767558 6693 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:29.767612 6693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:29.767760 6693 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:29.769503 6693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "e4cf7d189a3740e2a091648bf475454b"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "fb4d0ae8e8bcc9bbdf3e5b71b356c359"
server_key_iv: "24a947957bfb9037f2327a2bca8a15db"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.769944 6693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "e4cf7d189a3740e2a091648bf475454b"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "fb4d0ae8e8bcc9bbdf3e5b71b356c359"
server_key_iv: "24a947957bfb9037f2327a2bca8a15db"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.773150 6693 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:02:29.775318 6707 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:29.776275 6693 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251214 03:02:29.776369 6693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "e4cf7d189a3740e2a091648bf475454b"
format_stamp: "Formatted at 2025-12-14 03:02:29 on dist-test-slave-k7z7"
server_key: "fb4d0ae8e8bcc9bbdf3e5b71b356c359"
server_key_iv: "24a947957bfb9037f2327a2bca8a15db"
server_key_version: "encryptionkey@0"
I20251214 03:02:29.776465 6693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:29.786134 6693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:29.788704 6693 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:29.788872 6693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:29.789433 6693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:29.790289 6693 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:29.790336 6693 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:29.790375 6693 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:29.790408 6693 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:29.801913 6693 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:33661
I20251214 03:02:29.801928 6820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:33661 every 8 connection(s)
I20251214 03:02:29.802503 6693 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:02:29.809201 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 6693
I20251214 03:02:29.809324 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:02:29.809624 1655 external_mini_cluster.cc:1468] Setting key d16720c2c296e391f514715b997ce973
Dec 14 03:02:29 dist-test-slave-k7z7 krb5kdc[6312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681349, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:29.815308 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.804253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:51273 (local address 127.1.157.254:39061)
1214 03:02:29.804405 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.804409 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.805073 (+ 664us) server_negotiation.cc:408] Connection header received
1214 03:02:29.805959 (+ 886us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.805964 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.806026 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.806098 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:29.807762 (+ 1664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.808266 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.808909 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.809080 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.811968 (+ 2888us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:29.811983 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:29.811986 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:29.812015 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:29.813454 (+ 1439us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:29.813919 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:29.813922 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:29.813923 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:29.813979 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:29.814263 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:29.814266 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:29.814268 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:29.814442 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:29.814530 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.815050 (+ 520us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.815163 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251214 03:02:29.815907 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.804474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:39061 (local address 127.1.157.195:51273)
1214 03:02:29.804930 (+ 456us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:29.804966 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:29.805760 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:29.806276 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:29.806284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:29.806688 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:29.807598 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.807613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.808382 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.808385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:29.808799 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.808806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.809014 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.809929 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:29.809977 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:29.811828 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:29.813566 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:29.813572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:29.813582 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:29.813824 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:29.814056 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:29.814058 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:29.814060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:29.814145 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:29.814548 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:29.814553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:29.814800 (+ 247us) client_negotiation.cc:769] Sending connection context
1214 03:02:29.814974 (+ 174us) client_negotiation.cc:240] Negotiation successful
1214 03:02:29.815222 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":114,"threads_started":1}
I20251214 03:02:29.816952 6821 heartbeater.cc:344] Connected to a master server at 127.1.157.254:39061
I20251214 03:02:29.817237 6821 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:29.817713 6821 heartbeater.cc:507] Master 127.1.157.254:39061 requested a full tablet report, sending...
I20251214 03:02:29.818722 6359 ts_manager.cc:194] Registered new tserver with Master: e4cf7d189a3740e2a091648bf475454b (127.1.157.195:33661)
I20251214 03:02:29.819267 6359 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:51273
I20251214 03:02:29.823089 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251214 03:02:29.831266 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.824805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.1.157.254:39061)
1214 03:02:29.824918 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.824922 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.825035 (+ 113us) server_negotiation.cc:408] Connection header received
1214 03:02:29.825132 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.825134 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.825216 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.825279 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:29.826063 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.826557 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.827114 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.827254 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.828225 (+ 971us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:29.828239 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:29.828242 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:29.828265 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:29.829623 (+ 1358us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:29.829957 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:29.829959 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:29.829961 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:29.829996 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:29.830212 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:29.830215 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:29.830216 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:29.830414 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:29.830583 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.830964 (+ 381us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.831095 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251214 03:02:29.834026 6359 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:02:29.836416 6359 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:29.849472 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.845566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:33661 (local address 127.0.0.1:59612)
1214 03:02:29.846323 (+ 757us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:29.846335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:29.846411 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:29.846859 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:29.846862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:29.846880 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:29.847176 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.847183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.848267 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.848271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:29.849079 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.849086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.849211 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.849228 (+ 17us) client_negotiation.cc:769] Sending connection context
1214 03:02:29.849276 (+ 48us) client_negotiation.cc:240] Negotiation successful
1214 03:02:29.849328 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":104,"threads_started":1}
I20251214 03:02:29.849839 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.844855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:36123 (local address 127.0.0.1:58070)
1214 03:02:29.845425 (+ 570us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:29.845464 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:29.845617 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:29.846305 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:29.846311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:29.846333 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:29.846638 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.846646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.847847 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.847851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:29.848798 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.848807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.848909 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.848960 (+ 51us) client_negotiation.cc:769] Sending connection context
1214 03:02:29.849036 (+ 76us) client_negotiation.cc:240] Negotiation successful
1214 03:02:29.849123 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":70,"threads_started":1}
I20251214 03:02:29.850168 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.845602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.1.157.194:40119)
1214 03:02:29.845948 (+ 346us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.845954 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.846371 (+ 417us) server_negotiation.cc:408] Connection header received
1214 03:02:29.846469 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.846475 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.846634 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.846744 (+ 110us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:29.847323 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.848119 (+ 796us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.849257 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.849799 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.849898 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.849957 (+ 59us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.850039 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
I20251214 03:02:29.850178 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.844991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.1.157.193:36123)
1214 03:02:29.845359 (+ 368us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.845366 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.845555 (+ 189us) server_negotiation.cc:408] Connection header received
1214 03:02:29.845795 (+ 240us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.845800 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.845995 (+ 195us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.846123 (+ 128us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:29.846766 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.847733 (+ 967us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.849245 (+ 1512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.849773 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.849890 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.849958 (+ 68us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.850050 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":80,"threads_started":1}
I20251214 03:02:29.850404 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.845790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.1.157.195:33661)
1214 03:02:29.846189 (+ 399us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.846195 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.846493 (+ 298us) server_negotiation.cc:408] Connection header received
1214 03:02:29.846570 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.846576 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.846730 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.846857 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:29.847289 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.848159 (+ 870us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.849551 (+ 1392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.850038 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.850118 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.850192 (+ 74us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.850279 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":142,"threads_started":1}
I20251214 03:02:29.850395 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.844995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:40119 (local address 127.0.0.1:38198)
1214 03:02:29.846121 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:29.846134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:29.846215 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:29.846895 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:29.846898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:29.846912 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:29.847176 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.847183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.848232 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.848236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:29.848886 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.848896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.848982 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.849002 (+ 20us) client_negotiation.cc:769] Sending connection context
1214 03:02:29.849048 (+ 46us) client_negotiation.cc:240] Negotiation successful
1214 03:02:29.849125 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":117,"threads_started":1}
I20251214 03:02:29.853057 6485 tablet_service.cc:1505] Processing CreateTablet for tablet 351910e1c01841579e4dc7e7acd42632 (DEFAULT_TABLE table=test-table [id=48e407ff42f94c579e01f9494d00a25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:29.853061 6620 tablet_service.cc:1505] Processing CreateTablet for tablet 351910e1c01841579e4dc7e7acd42632 (DEFAULT_TABLE table=test-table [id=48e407ff42f94c579e01f9494d00a25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:29.853981 6485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351910e1c01841579e4dc7e7acd42632. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:29.853991 6620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351910e1c01841579e4dc7e7acd42632. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:29.853911 6755 tablet_service.cc:1505] Processing CreateTablet for tablet 351910e1c01841579e4dc7e7acd42632 (DEFAULT_TABLE table=test-table [id=48e407ff42f94c579e01f9494d00a25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:29.854691 6755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351910e1c01841579e4dc7e7acd42632. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:29.859376 6838 tablet_bootstrap.cc:492] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f: Bootstrap starting.
I20251214 03:02:29.860647 6840 tablet_bootstrap.cc:492] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: Bootstrap starting.
I20251214 03:02:29.860684 6839 tablet_bootstrap.cc:492] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: Bootstrap starting.
I20251214 03:02:29.861765 6838 tablet_bootstrap.cc:654] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:29.862602 6838 log.cc:826] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:29.862699 6839 tablet_bootstrap.cc:654] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:29.862766 6840 tablet_bootstrap.cc:654] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:29.863427 6840 log.cc:826] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:29.863466 6839 log.cc:826] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:29.864516 6838 tablet_bootstrap.cc:492] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f: No bootstrap required, opened a new log
I20251214 03:02:29.864701 6838 ts_tablet_manager.cc:1403] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251214 03:02:29.865080 6839 tablet_bootstrap.cc:492] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: No bootstrap required, opened a new log
I20251214 03:02:29.865080 6840 tablet_bootstrap.cc:492] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: No bootstrap required, opened a new log
I20251214 03:02:29.865280 6840 ts_tablet_manager.cc:1403] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251214 03:02:29.865280 6839 ts_tablet_manager.cc:1403] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251214 03:02:29.867700 6838 raft_consensus.cc:359] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36123 } } peers { permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 } } peers { permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 } }
I20251214 03:02:29.867884 6838 raft_consensus.cc:385] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:29.867955 6838 raft_consensus.cc:740] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75ec5a2ebb54786a5a1a41e890f9e2f, State: Initialized, Role: FOLLOWER
I20251214 03:02:29.868376 6838 consensus_queue.cc:260] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36123 } } peers { permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 } } peers { permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 } }
I20251214 03:02:29.868737 6840 raft_consensus.cc:359] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36123 } } peers { permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 } } peers { permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 } }
I20251214 03:02:29.868737 6839 raft_consensus.cc:359] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36123 } } peers { permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 } } peers { permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 } }
I20251214 03:02:29.868927 6839 raft_consensus.cc:385] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:29.868980 6839 raft_consensus.cc:740] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04cd16db5dab4ed1a1c858665eb81141, State: Initialized, Role: FOLLOWER
I20251214 03:02:29.869028 6551 heartbeater.cc:499] Master 127.1.157.254:39061 was elected leader, sending a full tablet report...
I20251214 03:02:29.869344 6838 ts_tablet_manager.cc:1434] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251214 03:02:29.869388 6839 consensus_queue.cc:260] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36123 } } peers { permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 } } peers { permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 } }
I20251214 03:02:29.869729 6840 raft_consensus.cc:385] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:29.869803 6840 raft_consensus.cc:740] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cf7d189a3740e2a091648bf475454b, State: Initialized, Role: FOLLOWER
I20251214 03:02:29.870162 6839 ts_tablet_manager.cc:1434] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251214 03:02:29.870191 6840 consensus_queue.cc:260] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36123 } } peers { permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 } } peers { permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 } }
I20251214 03:02:29.870514 6686 heartbeater.cc:499] Master 127.1.157.254:39061 was elected leader, sending a full tablet report...
I20251214 03:02:29.870987 6840 ts_tablet_manager.cc:1434] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251214 03:02:29.871306 6821 heartbeater.cc:499] Master 127.1.157.254:39061 was elected leader, sending a full tablet report...
I20251214 03:02:29.934671 6844 raft_consensus.cc:493] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:29.934851 6844 raft_consensus.cc:515] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36123 } } peers { permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 } } peers { permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 } }
I20251214 03:02:29.935988 6844 leader_election.cc:290] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04cd16db5dab4ed1a1c858665eb81141 (127.1.157.194:40119), e4cf7d189a3740e2a091648bf475454b (127.1.157.195:33661)
I20251214 03:02:29.939563 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.936222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:33661 (local address 127.1.157.193:46507)
1214 03:02:29.936639 (+ 417us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:29.936652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:29.936753 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:29.936986 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:29.936988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:29.937003 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:29.937315 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.937321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.938203 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.938207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:29.939099 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.939109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.939224 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.939246 (+ 22us) client_negotiation.cc:769] Sending connection context
1214 03:02:29.939295 (+ 49us) client_negotiation.cc:240] Negotiation successful
1214 03:02:29.939351 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":112,"threads_started":1}
I20251214 03:02:29.939564 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.936206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:40119 (local address 127.1.157.193:45801)
1214 03:02:29.936543 (+ 337us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:29.936554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:29.936651 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:29.936881 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:29.936885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:29.936899 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:29.937158 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.937165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.938132 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.938135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:29.939099 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:29.939109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.939228 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.939246 (+ 18us) client_negotiation.cc:769] Sending connection context
1214 03:02:29.939295 (+ 49us) client_negotiation.cc:240] Negotiation successful
1214 03:02:29.939351 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":100,"threads_started":1}
I20251214 03:02:29.940025 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.936302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:45801 (local address 127.1.157.194:40119)
1214 03:02:29.936442 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.936446 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.936628 (+ 182us) server_negotiation.cc:408] Connection header received
1214 03:02:29.936739 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.936743 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.936794 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.936864 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:29.937298 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.938026 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.939258 (+ 1232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.939782 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.939817 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.939864 (+ 47us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.939917 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251214 03:02:29.940025 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.936325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:46507 (local address 127.1.157.195:33661)
1214 03:02:29.936449 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.936453 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.936734 (+ 281us) server_negotiation.cc:408] Connection header received
1214 03:02:29.936838 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.936842 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.936892 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.936969 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:29.937424 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.938110 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.939249 (+ 1139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.939782 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.939817 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.939864 (+ 47us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.939917 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251214 03:02:29.940603 6640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "351910e1c01841579e4dc7e7acd42632" candidate_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04cd16db5dab4ed1a1c858665eb81141" is_pre_election: true
I20251214 03:02:29.940606 6775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "351910e1c01841579e4dc7e7acd42632" candidate_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cf7d189a3740e2a091648bf475454b" is_pre_election: true
I20251214 03:02:29.940833 6775 raft_consensus.cc:2468] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e75ec5a2ebb54786a5a1a41e890f9e2f in term 0.
I20251214 03:02:29.940834 6640 raft_consensus.cc:2468] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e75ec5a2ebb54786a5a1a41e890f9e2f in term 0.
I20251214 03:02:29.941280 6441 leader_election.cc:304] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4cf7d189a3740e2a091648bf475454b, e75ec5a2ebb54786a5a1a41e890f9e2f; no voters:
I20251214 03:02:29.941521 6844 raft_consensus.cc:2804] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:29.941593 6844 raft_consensus.cc:493] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:29.941639 6844 raft_consensus.cc:3060] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:29.942526 6844 raft_consensus.cc:515] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36123 } } peers { permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 } } peers { permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 } }
I20251214 03:02:29.942886 6844 leader_election.cc:290] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [CANDIDATE]: Term 1 election: Requested vote from peers 04cd16db5dab4ed1a1c858665eb81141 (127.1.157.194:40119), e4cf7d189a3740e2a091648bf475454b (127.1.157.195:33661)
I20251214 03:02:29.943305 6775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "351910e1c01841579e4dc7e7acd42632" candidate_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cf7d189a3740e2a091648bf475454b"
I20251214 03:02:29.943331 6640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "351910e1c01841579e4dc7e7acd42632" candidate_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04cd16db5dab4ed1a1c858665eb81141"
I20251214 03:02:29.943446 6775 raft_consensus.cc:3060] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:29.943459 6640 raft_consensus.cc:3060] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:29.944648 6640 raft_consensus.cc:2468] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e75ec5a2ebb54786a5a1a41e890f9e2f in term 1.
I20251214 03:02:29.944648 6775 raft_consensus.cc:2468] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e75ec5a2ebb54786a5a1a41e890f9e2f in term 1.
I20251214 03:02:29.944972 6439 leader_election.cc:304] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04cd16db5dab4ed1a1c858665eb81141, e75ec5a2ebb54786a5a1a41e890f9e2f; no voters:
I20251214 03:02:29.945155 6844 raft_consensus.cc:2804] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:29.945405 6844 raft_consensus.cc:697] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [term 1 LEADER]: Becoming Leader. State: Replica: e75ec5a2ebb54786a5a1a41e890f9e2f, State: Running, Role: LEADER
I20251214 03:02:29.945719 6844 consensus_queue.cc:237] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36123 } } peers { permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 } } peers { permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 } }
I20251214 03:02:29.948678 6358 catalog_manager.cc:5654] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f reported cstate change: term changed from 0 to 1, leader changed from <none> to e75ec5a2ebb54786a5a1a41e890f9e2f (127.1.157.193). New cstate: current_term: 1 leader_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75ec5a2ebb54786a5a1a41e890f9e2f" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 } health_report { overall_health: UNKNOWN } } }
I20251214 03:02:29.975610 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:29.972333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.1.157.193:36123)
1214 03:02:29.972541 (+ 208us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:29.972546 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:29.972560 (+ 14us) server_negotiation.cc:408] Connection header received
1214 03:02:29.972644 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:29.972647 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:29.972706 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:29.972799 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:29.973210 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.973721 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:29.974404 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:29.974602 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:29.974710 (+ 108us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:29.975192 (+ 482us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:29.975314 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:29.975398 (+ 84us) server_negotiation.cc:300] Negotiation successful
1214 03:02:29.975460 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20251214 03:02:29.981472 6640 raft_consensus.cc:1275] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141 [term 1 FOLLOWER]: Refusing update from remote peer e75ec5a2ebb54786a5a1a41e890f9e2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:29.981472 6775 raft_consensus.cc:1275] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b [term 1 FOLLOWER]: Refusing update from remote peer e75ec5a2ebb54786a5a1a41e890f9e2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:29.982156 6844 consensus_queue.cc:1048] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04cd16db5dab4ed1a1c858665eb81141" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:29.982352 6849 consensus_queue.cc:1048] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cf7d189a3740e2a091648bf475454b" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:29.990752 6853 mvcc.cc:204] Tried to move back new op lower bound from 7232230809516601344 to 7232230809380466688. Current Snapshot: MvccSnapshot[applied={T|T < 7232230809516601344}]
I20251214 03:02:29.999907 6358 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33370:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:30.000066 6358 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33370:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:30.002586 6358 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 238ec04c775e414a9a8a715bbc189638: Sending DeleteTablet for 3 replicas of tablet 351910e1c01841579e4dc7e7acd42632
I20251214 03:02:30.003440 6755 tablet_service.cc:1552] Processing DeleteTablet for tablet 351910e1c01841579e4dc7e7acd42632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:29 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:59612
I20251214 03:02:30.003437 6620 tablet_service.cc:1552] Processing DeleteTablet for tablet 351910e1c01841579e4dc7e7acd42632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:29 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:38198
I20251214 03:02:30.003993 6862 tablet_replica.cc:333] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: stopping tablet replica
I20251214 03:02:30.004007 6485 tablet_service.cc:1552] Processing DeleteTablet for tablet 351910e1c01841579e4dc7e7acd42632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:29 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:58070
I20251214 03:02:30.004279 6862 raft_consensus.cc:2243] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:30.004510 6864 tablet_replica.cc:333] T 351910e1c01841579e4dc7e7acd42632 P e75ec5a2ebb54786a5a1a41e890f9e2f: stopping tablet replica
I20251214 03:02:30.004637 6862 raft_consensus.cc:2272] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:30.004726 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 6423
I20251214 03:02:30.004997 6863 tablet_replica.cc:333] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: stopping tablet replica
I20251214 03:02:30.005311 6863 raft_consensus.cc:2243] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:30.005605 6863 raft_consensus.cc:2272] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:30.007011 6863 ts_tablet_manager.cc:1916] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:30.008978 6863 ts_tablet_manager.cc:1929] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:02:30.009058 6863 log.cc:1199] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/351910e1c01841579e4dc7e7acd42632
I20251214 03:02:30.009410 6863 ts_tablet_manager.cc:1950] T 351910e1c01841579e4dc7e7acd42632 P 04cd16db5dab4ed1a1c858665eb81141: Deleting consensus metadata
I20251214 03:02:30.010201 6862 ts_tablet_manager.cc:1916] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:30.010620 6344 catalog_manager.cc:4985] TS 04cd16db5dab4ed1a1c858665eb81141 (127.1.157.194:40119): tablet 351910e1c01841579e4dc7e7acd42632 (table test-table [id=48e407ff42f94c579e01f9494d00a25a]) successfully deleted
W20251214 03:02:30.010967 6345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.157.193:36123 (error 108)
I20251214 03:02:30.011255 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 6558
I20251214 03:02:30.012238 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.011946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:36123 (local address 127.0.0.1:58078)
1214 03:02:30.012082 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:30.012147 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.193:36123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20251214 03:02:30.012377 6345 catalog_manager.cc:4712] TS e75ec5a2ebb54786a5a1a41e890f9e2f (127.1.157.193:36123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 351910e1c01841579e4dc7e7acd42632: Network error: Client connection negotiation failed: client connection to 127.1.157.193:36123: connect: Connection refused (error 111)
I20251214 03:02:30.012755 6862 ts_tablet_manager.cc:1929] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:02:30.012830 6862 log.cc:1199] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/351910e1c01841579e4dc7e7acd42632
I20251214 03:02:30.013065 6862 ts_tablet_manager.cc:1950] T 351910e1c01841579e4dc7e7acd42632 P e4cf7d189a3740e2a091648bf475454b: Deleting consensus metadata
I20251214 03:02:30.014148 6346 catalog_manager.cc:4985] TS e4cf7d189a3740e2a091648bf475454b (127.1.157.195:33661): tablet 351910e1c01841579e4dc7e7acd42632 (table test-table [id=48e407ff42f94c579e01f9494d00a25a]) successfully deleted
I20251214 03:02:30.017398 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 6693
I20251214 03:02:30.022415 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 6328
2025-12-14T03:02:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3141 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-12-14T03:02:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:30Z Disabled control of system clock
I20251214 03:02:30.050289 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:43467
--webserver_interface=127.1.157.254
--webserver_port=0
--builtin_ntp_servers=127.1.157.212:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:43467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251214 03:02:30.155704 6869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:30.155936 6869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:30.155985 6869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:30.159327 6869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:30.159395 6869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:30.159411 6869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:30.159430 6869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:30.159453 6869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:30.163404 6869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:43467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:43467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:30.164429 6869 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:30.165560 6869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:30.170799 6875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:30.170799 6877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:30.170886 6874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:30.170984 6869 server_base.cc:1047] running on GCE node
I20251214 03:02:30.171518 6869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:30.172354 6869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:30.173514 6869 hybrid_clock.cc:648] HybridClock initialized: now 1765681350173492 us; error 35 us; skew 500 ppm
I20251214 03:02:30.175230 6869 webserver.cc:492] Webserver started at http://127.1.157.254:34771/ using document root <none> and password file <none>
I20251214 03:02:30.175745 6869 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:30.175799 6869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:30.175968 6869 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:30.177595 6869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "d9fb3f07331b4c5aa86b729f63760ca0"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "c0d4267f2ad962840e6ab153c685f054"
server_key_iv: "93b00706c1518e15a944d9506ad8e3b3"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.178035 6869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "d9fb3f07331b4c5aa86b729f63760ca0"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "c0d4267f2ad962840e6ab153c685f054"
server_key_iv: "93b00706c1518e15a944d9506ad8e3b3"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.181265 6869 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:02:30.183362 6883 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:30.184275 6869 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:30.184367 6869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "d9fb3f07331b4c5aa86b729f63760ca0"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "c0d4267f2ad962840e6ab153c685f054"
server_key_iv: "93b00706c1518e15a944d9506ad8e3b3"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.184453 6869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:30.188848 6869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:30.189371 6869 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:30.189507 6869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:30.195845 6935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:43467 every 8 connection(s)
I20251214 03:02:30.195855 6869 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:43467
I20251214 03:02:30.196827 6869 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:30.199409 6936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:30.204536 6936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0: Bootstrap starting.
I20251214 03:02:30.205307 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 6869
I20251214 03:02:30.205405 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:30.205636 1655 external_mini_cluster.cc:1468] Setting key eafe0c5500f348ae24409b79ecafda7e
I20251214 03:02:30.206945 6936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:30.207626 6936 log.cc:826] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:30.209599 6936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0: No bootstrap required, opened a new log
I20251214 03:02:30.210323 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.206655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.1.157.254:43467)
1214 03:02:30.207291 (+ 636us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.207299 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.207331 (+ 32us) server_negotiation.cc:408] Connection header received
1214 03:02:30.207878 (+ 547us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.207897 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.208211 (+ 314us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.208528 (+ 317us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.209311 (+ 783us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.209316 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.209325 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.209335 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.209375 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.209386 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.209481 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.209599 (+ 118us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.209806 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":138,"threads_started":1}
I20251214 03:02:30.212345 6936 raft_consensus.cc:359] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb3f07331b4c5aa86b729f63760ca0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43467 } }
I20251214 03:02:30.212527 6936 raft_consensus.cc:385] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:30.212563 6936 raft_consensus.cc:740] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fb3f07331b4c5aa86b729f63760ca0, State: Initialized, Role: FOLLOWER
I20251214 03:02:30.212944 6936 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb3f07331b4c5aa86b729f63760ca0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43467 } }
I20251214 03:02:30.213056 6936 raft_consensus.cc:399] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:30.213110 6936 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:30.213212 6936 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:30.213980 6936 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb3f07331b4c5aa86b729f63760ca0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43467 } }
I20251214 03:02:30.214298 6936 leader_election.cc:304] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9fb3f07331b4c5aa86b729f63760ca0; no voters:
I20251214 03:02:30.214558 6936 leader_election.cc:290] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:30.214699 6941 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:30.214881 6941 raft_consensus.cc:697] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [term 1 LEADER]: Becoming Leader. State: Replica: d9fb3f07331b4c5aa86b729f63760ca0, State: Running, Role: LEADER
I20251214 03:02:30.215153 6941 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb3f07331b4c5aa86b729f63760ca0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43467 } }
I20251214 03:02:30.215624 6936 sys_catalog.cc:565] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:30.216879 6942 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9fb3f07331b4c5aa86b729f63760ca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb3f07331b4c5aa86b729f63760ca0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43467 } } }
I20251214 03:02:30.216892 6943 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9fb3f07331b4c5aa86b729f63760ca0. Latest consensus state: current_term: 1 leader_uuid: "d9fb3f07331b4c5aa86b729f63760ca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fb3f07331b4c5aa86b729f63760ca0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43467 } } }
I20251214 03:02:30.217097 6942 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:30.217118 6943 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:30.217846 6950 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:30.220675 6950 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:30.225420 6950 catalog_manager.cc:1357] Generated new cluster ID: f156c81460c24b7baa8c07a81c227481
I20251214 03:02:30.225503 6950 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:30.242841 6950 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:30.243201 6950 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:30.250769 6950 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0: Generated new TSK 0
I20251214 03:02:30.251717 6950 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 03:02:30.259177 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:43467
--builtin_ntp_servers=127.1.157.212:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251214 03:02:30.364668 6960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:30.364902 6960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:30.364972 6960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:30.368279 6960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:30.368335 6960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:30.368429 6960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:30.372294 6960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:43467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:30.373340 6960 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:30.374445 6960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:30.380782 6966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:30.380764 6965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:30.380764 6968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:30.381045 6960 server_base.cc:1047] running on GCE node
I20251214 03:02:30.381440 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:30.382030 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:30.383211 6960 hybrid_clock.cc:648] HybridClock initialized: now 1765681350383176 us; error 46 us; skew 500 ppm
I20251214 03:02:30.384894 6960 webserver.cc:492] Webserver started at http://127.1.157.193:33351/ using document root <none> and password file <none>
I20251214 03:02:30.385438 6960 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:30.385490 6960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:30.385643 6960 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:30.387197 6960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "87b0001b4ad54fd09729af88ea28895e"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "eb4624e4a7c0b3ba43fba678dc0a0347"
server_key_iv: "fd63480e066a291094d9b80739353c2f"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.387609 6960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "87b0001b4ad54fd09729af88ea28895e"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "eb4624e4a7c0b3ba43fba678dc0a0347"
server_key_iv: "fd63480e066a291094d9b80739353c2f"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.390758 6960 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20251214 03:02:30.392901 6974 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:30.393901 6960 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:30.393988 6960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "87b0001b4ad54fd09729af88ea28895e"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "eb4624e4a7c0b3ba43fba678dc0a0347"
server_key_iv: "fd63480e066a291094d9b80739353c2f"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.394073 6960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:30.398515 6960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:30.399037 6960 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:30.399174 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:30.399654 6960 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:30.400527 6960 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:30.400571 6960 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:30.400621 6960 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:30.400648 6960 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:30.408838 6960 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:43457
I20251214 03:02:30.408869 7087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:43457 every 8 connection(s)
I20251214 03:02:30.409761 6960 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:02:30.414482 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 6960
I20251214 03:02:30.414603 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:02:30.414899 1655 external_mini_cluster.cc:1468] Setting key c16c0ece8dea999069d18c52f620296d
I20251214 03:02:30.415119 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.411480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:34023 (local address 127.1.157.254:43467)
1214 03:02:30.411610 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.411613 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.412479 (+ 866us) server_negotiation.cc:408] Connection header received
1214 03:02:30.413266 (+ 787us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.413270 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.413315 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.413395 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.414571 (+ 1176us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.414575 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.414577 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.414592 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.414620 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.414630 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.414731 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.414931 (+ 200us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.414986 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251214 03:02:30.415925 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.411716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:43467 (local address 127.1.157.193:34023)
1214 03:02:30.412335 (+ 619us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:30.412368 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:30.413046 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:30.413549 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:30.413557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:30.413650 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:30.414250 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:02:30.414252 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:30.414298 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:02:30.414371 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:30.414720 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:30.414730 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:30.414736 (+ 6us) client_negotiation.cc:769] Sending connection context
1214 03:02:30.414964 (+ 228us) client_negotiation.cc:240] Negotiation successful
1214 03:02:30.415205 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20251214 03:02:30.417323 7088 heartbeater.cc:344] Connected to a master server at 127.1.157.254:43467
I20251214 03:02:30.417412 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:43467
--builtin_ntp_servers=127.1.157.212:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251214 03:02:30.417627 7088 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:30.418081 7088 heartbeater.cc:507] Master 127.1.157.254:43467 requested a full tablet report, sending...
I20251214 03:02:30.419580 6900 ts_manager.cc:194] Registered new tserver with Master: 87b0001b4ad54fd09729af88ea28895e (127.1.157.193:43457)
W20251214 03:02:30.533386 7091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:30.533607 7091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:30.533677 7091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:30.536896 7091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:30.536950 7091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:30.537045 7091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:30.540756 7091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:43467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:30.541800 7091 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:30.542850 7091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:30.548842 7097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:30.548873 7099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:30.548868 7096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:30.549247 7091 server_base.cc:1047] running on GCE node
I20251214 03:02:30.549580 7091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:30.550064 7091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:30.551220 7091 hybrid_clock.cc:648] HybridClock initialized: now 1765681350551199 us; error 33 us; skew 500 ppm
I20251214 03:02:30.552983 7091 webserver.cc:492] Webserver started at http://127.1.157.194:40111/ using document root <none> and password file <none>
I20251214 03:02:30.553582 7091 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:30.553639 7091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:30.553920 7091 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:30.555557 7091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "0e74af9a68b3486186d76b921c3d68a8"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "714928400585625e72d131fe98e07522"
server_key_iv: "1e01a1377339b4fb8a0172e0d68e5677"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.555994 7091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "0e74af9a68b3486186d76b921c3d68a8"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "714928400585625e72d131fe98e07522"
server_key_iv: "1e01a1377339b4fb8a0172e0d68e5677"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.559093 7091 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:02:30.561112 7105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:30.562055 7091 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:30.562150 7091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "0e74af9a68b3486186d76b921c3d68a8"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "714928400585625e72d131fe98e07522"
server_key_iv: "1e01a1377339b4fb8a0172e0d68e5677"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.562258 7091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:30.566628 7091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:30.567142 7091 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:30.567273 7091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:30.567709 7091 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:30.568558 7091 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:30.568603 7091 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:30.568635 7091 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:30.568655 7091 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:30.576610 7091 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:41805
I20251214 03:02:30.576648 7218 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:41805 every 8 connection(s)
I20251214 03:02:30.577467 7091 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:02:30.582739 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 7091
I20251214 03:02:30.582855 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:02:30.582864 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.579060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:38705 (local address 127.1.157.254:43467)
1214 03:02:30.579228 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.579232 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.580254 (+ 1022us) server_negotiation.cc:408] Connection header received
1214 03:02:30.581030 (+ 776us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.581034 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.581086 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.581202 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.582354 (+ 1152us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.582365 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.582368 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.582384 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.582408 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.582420 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.582533 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.582686 (+ 153us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.582730 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251214 03:02:30.583124 1655 external_mini_cluster.cc:1468] Setting key 5b63026a2faf487458fb1bd4b2ca5f08
I20251214 03:02:30.583560 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.579305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:43467 (local address 127.1.157.194:38705)
1214 03:02:30.580072 (+ 767us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:30.580116 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:30.580834 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:30.581356 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:30.581364 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:30.581452 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:30.582025 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:02:30.582027 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:30.582058 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:02:30.582116 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:30.582546 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:30.582555 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:30.582561 (+ 6us) client_negotiation.cc:769] Sending connection context
1214 03:02:30.582746 (+ 185us) client_negotiation.cc:240] Negotiation successful
1214 03:02:30.582946 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":100,"threads_started":1}
I20251214 03:02:30.584509 7219 heartbeater.cc:344] Connected to a master server at 127.1.157.254:43467
I20251214 03:02:30.584746 7219 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:30.585137 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:43467
--builtin_ntp_servers=127.1.157.212:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251214 03:02:30.585209 7219 heartbeater.cc:507] Master 127.1.157.254:43467 requested a full tablet report, sending...
I20251214 03:02:30.586246 6900 ts_manager.cc:194] Registered new tserver with Master: 0e74af9a68b3486186d76b921c3d68a8 (127.1.157.194:41805)
W20251214 03:02:30.692138 7222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:30.692399 7222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:30.692454 7222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:30.695900 7222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:30.695962 7222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:30.696048 7222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:30.699908 7222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:43467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:30.700943 7222 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:30.702030 7222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:30.708249 7228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:30.708245 7230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:30.708271 7227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:30.708621 7222 server_base.cc:1047] running on GCE node
I20251214 03:02:30.708956 7222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:30.709504 7222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:30.710647 7222 hybrid_clock.cc:648] HybridClock initialized: now 1765681350710636 us; error 27 us; skew 500 ppm
I20251214 03:02:30.712266 7222 webserver.cc:492] Webserver started at http://127.1.157.195:34519/ using document root <none> and password file <none>
I20251214 03:02:30.712786 7222 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:30.712841 7222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:30.713007 7222 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:30.714627 7222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "7b1a758d49214e33aa2f09c665eb8267"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "bbd3b3245e1da7ae430555af515b10a0"
server_key_iv: "379efbcc90e4f6d1227cb0e8494673f9"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.715075 7222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "7b1a758d49214e33aa2f09c665eb8267"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "bbd3b3245e1da7ae430555af515b10a0"
server_key_iv: "379efbcc90e4f6d1227cb0e8494673f9"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.718253 7222 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251214 03:02:30.720259 7236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:30.721210 7222 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251214 03:02:30.721304 7222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "7b1a758d49214e33aa2f09c665eb8267"
format_stamp: "Formatted at 2025-12-14 03:02:30 on dist-test-slave-k7z7"
server_key: "bbd3b3245e1da7ae430555af515b10a0"
server_key_iv: "379efbcc90e4f6d1227cb0e8494673f9"
server_key_version: "encryptionkey@0"
I20251214 03:02:30.721381 7222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:30.725740 7222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:30.726276 7222 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:30.726419 7222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:30.726889 7222 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:30.727766 7222 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:30.727811 7222 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:30.727860 7222 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:30.727885 7222 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:30.736343 7222 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:45711
I20251214 03:02:30.736409 7349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:45711 every 8 connection(s)
I20251214 03:02:30.737339 7222 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:02:30.740541 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 7222
I20251214 03:02:30.740662 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:02:30.740916 1655 external_mini_cluster.cc:1468] Setting key 91f9990e74378d84692f7f857b713a8a
I20251214 03:02:30.742990 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.739117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:54351 (local address 127.1.157.254:43467)
1214 03:02:30.739265 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.739270 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.740177 (+ 907us) server_negotiation.cc:408] Connection header received
1214 03:02:30.741251 (+ 1074us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.741255 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.741307 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.741405 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.742566 (+ 1161us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.742570 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.742573 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.742586 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.742610 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.742620 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.742712 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.742819 (+ 107us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.742862 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251214 03:02:30.743716 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.739427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:43467 (local address 127.1.157.195:54351)
1214 03:02:30.740042 (+ 615us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:30.740076 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:30.740934 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:30.741567 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:30.741578 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:30.741675 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:30.742346 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:02:30.742348 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:30.742379 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:02:30.742434 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:30.742696 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:30.742700 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:30.742704 (+ 4us) client_negotiation.cc:769] Sending connection context
1214 03:02:30.742879 (+ 175us) client_negotiation.cc:240] Negotiation successful
1214 03:02:30.743079 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20251214 03:02:30.744686 7350 heartbeater.cc:344] Connected to a master server at 127.1.157.254:43467
I20251214 03:02:30.744901 7350 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:30.745326 7350 heartbeater.cc:507] Master 127.1.157.254:43467 requested a full tablet report, sending...
I20251214 03:02:30.746290 6900 ts_manager.cc:194] Registered new tserver with Master: 7b1a758d49214e33aa2f09c665eb8267 (127.1.157.195:45711)
I20251214 03:02:30.754621 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251214 03:02:30.757711 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.756295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.1.157.254:43467)
1214 03:02:30.756411 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.756415 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.756523 (+ 108us) server_negotiation.cc:408] Connection header received
1214 03:02:30.756618 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.756620 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.756655 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.756718 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.757379 (+ 661us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.757382 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.757384 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.757393 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.757410 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.757417 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.757482 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.757584 (+ 102us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.757613 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251214 03:02:30.760041 6900 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:02:30.762482 6900 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:30.773371 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.770762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.1.157.193:43457)
1214 03:02:30.771177 (+ 415us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.771183 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.771219 (+ 36us) server_negotiation.cc:408] Connection header received
1214 03:02:30.771276 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.771281 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.771445 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.771579 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.772795 (+ 1216us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.772801 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.772810 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.772823 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.772879 (+ 56us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.772894 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.773002 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.773132 (+ 130us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.773222 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":95,"threads_started":1}
I20251214 03:02:30.773360 7363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.770243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:43457 (local address 127.0.0.1:47922)
1214 03:02:30.771090 (+ 847us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:30.771102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:30.771165 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:30.771620 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:30.771626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:30.771692 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:30.772303 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:02:30.772306 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:30.772343 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:02:30.772665 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:30.772983 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:30.772993 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:30.772999 (+ 6us) client_negotiation.cc:769] Sending connection context
1214 03:02:30.773105 (+ 106us) client_negotiation.cc:240] Negotiation successful
1214 03:02:30.773198 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":90,"threads_started":1}
I20251214 03:02:30.773360 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.770048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:45711 (local address 127.0.0.1:41562)
1214 03:02:30.770781 (+ 733us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:30.770818 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:30.770919 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:30.772223 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:30.772227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:30.772243 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:30.772673 (+ 430us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:02:30.772675 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:30.772683 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:02:30.772703 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:30.773057 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:30.773060 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:30.773062 (+ 2us) client_negotiation.cc:769] Sending connection context
1214 03:02:30.773139 (+ 77us) client_negotiation.cc:240] Negotiation successful
1214 03:02:30.773194 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":71,"threads_started":1}
I20251214 03:02:30.774091 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.770743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:41805 (local address 127.0.0.1:34912)
1214 03:02:30.771494 (+ 751us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:30.771506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:30.771572 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:30.772685 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:30.772688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:30.772702 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:30.773522 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:02:30.773524 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:30.773536 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:02:30.773562 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:30.773866 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:30.773870 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:30.773873 (+ 3us) client_negotiation.cc:769] Sending connection context
1214 03:02:30.773952 (+ 79us) client_negotiation.cc:240] Negotiation successful
1214 03:02:30.773982 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":86,"threads_started":1}
I20251214 03:02:30.774267 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.771121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.1.157.194:41805)
1214 03:02:30.771791 (+ 670us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.771797 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.771820 (+ 23us) server_negotiation.cc:408] Connection header received
1214 03:02:30.771870 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.771875 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.772011 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.772135 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.773687 (+ 1552us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.773692 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.773701 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.773712 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.773766 (+ 54us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.773780 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.773883 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.773960 (+ 77us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.774020 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":127,"threads_started":1}
I20251214 03:02:30.774485 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.770186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.1.157.195:45711)
1214 03:02:30.771396 (+ 1210us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.771401 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.771421 (+ 20us) server_negotiation.cc:408] Connection header received
1214 03:02:30.771481 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.771486 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.771615 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.771727 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.772825 (+ 1098us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.772830 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.772837 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.772847 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.772890 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.772901 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.773002 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.773205 (+ 203us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.774290 (+ 1085us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":106,"threads_started":1}
I20251214 03:02:30.776427 7153 tablet_service.cc:1505] Processing CreateTablet for tablet 174de7979ec347aa8445fcb3204e82b1 (DEFAULT_TABLE table=test-table [id=b54b12b7524c409291f194ac68659eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:30.776325 7022 tablet_service.cc:1505] Processing CreateTablet for tablet 174de7979ec347aa8445fcb3204e82b1 (DEFAULT_TABLE table=test-table [id=b54b12b7524c409291f194ac68659eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:30.776822 7284 tablet_service.cc:1505] Processing CreateTablet for tablet 174de7979ec347aa8445fcb3204e82b1 (DEFAULT_TABLE table=test-table [id=b54b12b7524c409291f194ac68659eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:30.777320 7153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 174de7979ec347aa8445fcb3204e82b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:30.777320 7022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 174de7979ec347aa8445fcb3204e82b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:30.777596 7284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 174de7979ec347aa8445fcb3204e82b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:30.783236 7367 tablet_bootstrap.cc:492] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: Bootstrap starting.
I20251214 03:02:30.784184 7369 tablet_bootstrap.cc:492] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: Bootstrap starting.
I20251214 03:02:30.785696 7367 tablet_bootstrap.cc:654] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:30.785907 7369 tablet_bootstrap.cc:654] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:30.786548 7367 log.cc:826] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:30.786589 7369 log.cc:826] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:30.787288 7368 tablet_bootstrap.cc:492] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e: Bootstrap starting.
I20251214 03:02:30.788487 7369 tablet_bootstrap.cc:492] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: No bootstrap required, opened a new log
I20251214 03:02:30.788487 7367 tablet_bootstrap.cc:492] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: No bootstrap required, opened a new log
I20251214 03:02:30.788681 7369 ts_tablet_manager.cc:1403] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251214 03:02:30.788681 7367 ts_tablet_manager.cc:1403] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251214 03:02:30.789139 7368 tablet_bootstrap.cc:654] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:30.789855 7368 log.cc:826] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:30.791493 7368 tablet_bootstrap.cc:492] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e: No bootstrap required, opened a new log
I20251214 03:02:30.791754 7368 ts_tablet_manager.cc:1403] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251214 03:02:30.791860 7367 raft_consensus.cc:359] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 } } peers { permanent_uuid: "87b0001b4ad54fd09729af88ea28895e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43457 } } peers { permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 } }
I20251214 03:02:30.791860 7369 raft_consensus.cc:359] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 } } peers { permanent_uuid: "87b0001b4ad54fd09729af88ea28895e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43457 } } peers { permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 } }
I20251214 03:02:30.792073 7369 raft_consensus.cc:385] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:30.792073 7367 raft_consensus.cc:385] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:30.792122 7369 raft_consensus.cc:740] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1a758d49214e33aa2f09c665eb8267, State: Initialized, Role: FOLLOWER
I20251214 03:02:30.792122 7367 raft_consensus.cc:740] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e74af9a68b3486186d76b921c3d68a8, State: Initialized, Role: FOLLOWER
I20251214 03:02:30.792634 7367 consensus_queue.cc:260] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 } } peers { permanent_uuid: "87b0001b4ad54fd09729af88ea28895e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43457 } } peers { permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 } }
I20251214 03:02:30.792634 7369 consensus_queue.cc:260] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 } } peers { permanent_uuid: "87b0001b4ad54fd09729af88ea28895e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43457 } } peers { permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 } }
I20251214 03:02:30.793606 7367 ts_tablet_manager.cc:1434] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251214 03:02:30.793606 7369 ts_tablet_manager.cc:1434] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251214 03:02:30.793931 7350 heartbeater.cc:499] Master 127.1.157.254:43467 was elected leader, sending a full tablet report...
I20251214 03:02:30.793937 7219 heartbeater.cc:499] Master 127.1.157.254:43467 was elected leader, sending a full tablet report...
I20251214 03:02:30.794916 7368 raft_consensus.cc:359] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 } } peers { permanent_uuid: "87b0001b4ad54fd09729af88ea28895e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43457 } } peers { permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 } }
I20251214 03:02:30.795969 7368 raft_consensus.cc:385] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:30.796049 7368 raft_consensus.cc:740] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b0001b4ad54fd09729af88ea28895e, State: Initialized, Role: FOLLOWER
I20251214 03:02:30.796489 7368 consensus_queue.cc:260] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 } } peers { permanent_uuid: "87b0001b4ad54fd09729af88ea28895e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43457 } } peers { permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 } }
I20251214 03:02:30.797422 7088 heartbeater.cc:499] Master 127.1.157.254:43467 was elected leader, sending a full tablet report...
I20251214 03:02:30.798768 7368 ts_tablet_manager.cc:1434] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20251214 03:02:30.828267 7220 tablet.cc:2378] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 03:02:30.887814 7375 raft_consensus.cc:493] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:30.887997 7375 raft_consensus.cc:515] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 } } peers { permanent_uuid: "87b0001b4ad54fd09729af88ea28895e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43457 } } peers { permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 } }
I20251214 03:02:30.889217 7375 leader_election.cc:290] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b1a758d49214e33aa2f09c665eb8267 (127.1.157.195:45711), 0e74af9a68b3486186d76b921c3d68a8 (127.1.157.194:41805)
I20251214 03:02:30.891069 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.889488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:45711 (local address 127.1.157.193:52157)
1214 03:02:30.889617 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:30.889633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:30.889736 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:30.889970 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:30.889973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:30.889990 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:30.890494 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:02:30.890497 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:30.890510 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:02:30.890532 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:30.890811 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:30.890814 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:30.890817 (+ 3us) client_negotiation.cc:769] Sending connection context
1214 03:02:30.890899 (+ 82us) client_negotiation.cc:240] Negotiation successful
1214 03:02:30.890934 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20251214 03:02:30.891088 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.889592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:52157 (local address 127.1.157.195:45711)
1214 03:02:30.889722 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.889727 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.889742 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:02:30.889831 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.889834 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.889869 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.889934 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.890663 (+ 729us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.890667 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.890671 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.890684 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.890709 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.890719 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.890800 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.890896 (+ 96us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.890939 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251214 03:02:30.891716 7304 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174de7979ec347aa8445fcb3204e82b1" candidate_uuid: "87b0001b4ad54fd09729af88ea28895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1a758d49214e33aa2f09c665eb8267" is_pre_election: true
I20251214 03:02:30.891894 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.889585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:43007 (local address 127.1.157.194:41805)
1214 03:02:30.889737 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.889742 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.890006 (+ 264us) server_negotiation.cc:408] Connection header received
1214 03:02:30.890135 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.890138 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.890177 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.890260 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.891564 (+ 1304us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.891569 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.891571 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.891583 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.891602 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.891610 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.891701 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.891751 (+ 50us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.891786 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251214 03:02:30.891927 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.889488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:41805 (local address 127.1.157.193:43007)
1214 03:02:30.889924 (+ 436us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:30.889935 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:30.890020 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:30.890302 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:30.890305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:30.890331 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:30.891403 (+ 1072us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:02:30.891405 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:30.891418 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:02:30.891448 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:30.891691 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:30.891694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:30.891696 (+ 2us) client_negotiation.cc:769] Sending connection context
1214 03:02:30.891751 (+ 55us) client_negotiation.cc:240] Negotiation successful
1214 03:02:30.891778 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":114,"thread_start_us":147,"threads_started":1}
I20251214 03:02:30.892001 7304 raft_consensus.cc:2468] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87b0001b4ad54fd09729af88ea28895e in term 0.
I20251214 03:02:30.892504 6976 leader_election.cc:304] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1a758d49214e33aa2f09c665eb8267, 87b0001b4ad54fd09729af88ea28895e; no voters:
I20251214 03:02:30.892624 7173 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174de7979ec347aa8445fcb3204e82b1" candidate_uuid: "87b0001b4ad54fd09729af88ea28895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e74af9a68b3486186d76b921c3d68a8" is_pre_election: true
I20251214 03:02:30.892735 7375 raft_consensus.cc:2804] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:30.892807 7375 raft_consensus.cc:493] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:30.892848 7375 raft_consensus.cc:3060] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:30.892863 7173 raft_consensus.cc:2468] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87b0001b4ad54fd09729af88ea28895e in term 0.
I20251214 03:02:30.893838 7375 raft_consensus.cc:515] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 } } peers { permanent_uuid: "87b0001b4ad54fd09729af88ea28895e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43457 } } peers { permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 } }
I20251214 03:02:30.894182 7375 leader_election.cc:290] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [CANDIDATE]: Term 1 election: Requested vote from peers 7b1a758d49214e33aa2f09c665eb8267 (127.1.157.195:45711), 0e74af9a68b3486186d76b921c3d68a8 (127.1.157.194:41805)
I20251214 03:02:30.894510 7304 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174de7979ec347aa8445fcb3204e82b1" candidate_uuid: "87b0001b4ad54fd09729af88ea28895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1a758d49214e33aa2f09c665eb8267"
I20251214 03:02:30.894568 7173 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174de7979ec347aa8445fcb3204e82b1" candidate_uuid: "87b0001b4ad54fd09729af88ea28895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e74af9a68b3486186d76b921c3d68a8"
I20251214 03:02:30.894611 7304 raft_consensus.cc:3060] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:30.894690 7173 raft_consensus.cc:3060] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:30.895864 7304 raft_consensus.cc:2468] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b0001b4ad54fd09729af88ea28895e in term 1.
I20251214 03:02:30.896041 7173 raft_consensus.cc:2468] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b0001b4ad54fd09729af88ea28895e in term 1.
I20251214 03:02:30.896152 6976 leader_election.cc:304] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1a758d49214e33aa2f09c665eb8267, 87b0001b4ad54fd09729af88ea28895e; no voters:
I20251214 03:02:30.896384 7375 raft_consensus.cc:2804] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:30.896554 7375 raft_consensus.cc:697] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [term 1 LEADER]: Becoming Leader. State: Replica: 87b0001b4ad54fd09729af88ea28895e, State: Running, Role: LEADER
I20251214 03:02:30.896875 7375 consensus_queue.cc:237] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 } } peers { permanent_uuid: "87b0001b4ad54fd09729af88ea28895e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43457 } } peers { permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 } }
I20251214 03:02:30.900053 6900 catalog_manager.cc:5654] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e reported cstate change: term changed from 0 to 1, leader changed from <none> to 87b0001b4ad54fd09729af88ea28895e (127.1.157.193). New cstate: current_term: 1 leader_uuid: "87b0001b4ad54fd09729af88ea28895e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87b0001b4ad54fd09729af88ea28895e" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 43457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 } health_report { overall_health: UNKNOWN } } }
W20251214 03:02:30.910818 7089 tablet.cc:2378] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 03:02:30.926051 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.924456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.1.157.193:43457)
1214 03:02:30.924602 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:30.924607 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:30.924625 (+ 18us) server_negotiation.cc:408] Connection header received
1214 03:02:30.924709 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:30.924712 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:30.924796 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:30.924888 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:30.925680 (+ 792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:30.925684 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:30.925687 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:30.925700 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:30.925721 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:30.925730 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:30.925805 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:30.925882 (+ 77us) server_negotiation.cc:300] Negotiation successful
1214 03:02:30.925920 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251214 03:02:30.931134 7304 raft_consensus.cc:1275] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267 [term 1 FOLLOWER]: Refusing update from remote peer 87b0001b4ad54fd09729af88ea28895e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:30.931134 7173 raft_consensus.cc:1275] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8 [term 1 FOLLOWER]: Refusing update from remote peer 87b0001b4ad54fd09729af88ea28895e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:30.931753 7377 consensus_queue.cc:1048] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b1a758d49214e33aa2f09c665eb8267" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 45711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:30.931900 7375 consensus_queue.cc:1048] T 174de7979ec347aa8445fcb3204e82b1 P 87b0001b4ad54fd09729af88ea28895e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e74af9a68b3486186d76b921c3d68a8" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:30.940232 7381 mvcc.cc:204] Tried to move back new op lower bound from 7232230813407645696 to 7232230813276655616. Current Snapshot: MvccSnapshot[applied={T|T < 7232230813407645696 or (T in {7232230813407645696})}]
I20251214 03:02:30.945458 6899 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:30.945607 6899 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:30.947929 6899 catalog_manager.cc:5941] T 00000000000000000000000000000000 P d9fb3f07331b4c5aa86b729f63760ca0: Sending DeleteTablet for 3 replicas of tablet 174de7979ec347aa8445fcb3204e82b1
I20251214 03:02:30.948686 7284 tablet_service.cc:1552] Processing DeleteTablet for tablet 174de7979ec347aa8445fcb3204e82b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:30 UTC) from {username='slave'} at 127.0.0.1:41562
I20251214 03:02:30.949095 7153 tablet_service.cc:1552] Processing DeleteTablet for tablet 174de7979ec347aa8445fcb3204e82b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:30 UTC) from {username='slave'} at 127.0.0.1:34912
I20251214 03:02:30.949234 7022 tablet_service.cc:1552] Processing DeleteTablet for tablet 174de7979ec347aa8445fcb3204e82b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:30 UTC) from {username='slave'} at 127.0.0.1:47922
I20251214 03:02:30.949690 7389 tablet_replica.cc:333] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: stopping tablet replica
I20251214 03:02:30.949858 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 6960
I20251214 03:02:30.949885 7390 tablet_replica.cc:333] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: stopping tablet replica
I20251214 03:02:30.950564 7389 raft_consensus.cc:2243] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:30.950860 7389 raft_consensus.cc:2272] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:30.951802 7390 raft_consensus.cc:2243] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:30.952095 7390 raft_consensus.cc:2272] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:30.952375 7389 ts_tablet_manager.cc:1916] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:30.953522 7390 ts_tablet_manager.cc:1916] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:30.954871 7389 ts_tablet_manager.cc:1929] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:02:30.954967 7389 log.cc:1199] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/174de7979ec347aa8445fcb3204e82b1
I20251214 03:02:30.955273 7389 ts_tablet_manager.cc:1950] T 174de7979ec347aa8445fcb3204e82b1 P 7b1a758d49214e33aa2f09c665eb8267: Deleting consensus metadata
I20251214 03:02:30.955525 7390 ts_tablet_manager.cc:1929] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:02:30.955601 7390 log.cc:1199] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/174de7979ec347aa8445fcb3204e82b1
I20251214 03:02:30.955852 7390 ts_tablet_manager.cc:1950] T 174de7979ec347aa8445fcb3204e82b1 P 0e74af9a68b3486186d76b921c3d68a8: Deleting consensus metadata
I20251214 03:02:30.956458 6885 catalog_manager.cc:4985] TS 7b1a758d49214e33aa2f09c665eb8267 (127.1.157.195:45711): tablet 174de7979ec347aa8445fcb3204e82b1 (table test-table [id=b54b12b7524c409291f194ac68659eb0]) successfully deleted
I20251214 03:02:30.956799 6884 catalog_manager.cc:4985] TS 0e74af9a68b3486186d76b921c3d68a8 (127.1.157.194:41805): tablet 174de7979ec347aa8445fcb3204e82b1 (table test-table [id=b54b12b7524c409291f194ac68659eb0]) successfully deleted
W20251214 03:02:30.957062 6886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.157.193:43457 (error 108)
I20251214 03:02:30.957312 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 7091
I20251214 03:02:30.959070 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:30.958785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:43457 (local address 127.0.0.1:47954)
1214 03:02:30.958917 (+ 132us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:30.958982 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.193:43457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20251214 03:02:30.959198 6886 catalog_manager.cc:4712] TS 87b0001b4ad54fd09729af88ea28895e (127.1.157.193:43457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 174de7979ec347aa8445fcb3204e82b1: Network error: Client connection negotiation failed: client connection to 127.1.157.193:43457: connect: Connection refused (error 111)
I20251214 03:02:30.962778 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 7222
I20251214 03:02:30.967569 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 6869
2025-12-14T03:02:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (944 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251214 03:02:30.987761 1655 mini_oidc.cc:150] Starting JWKS server
I20251214 03:02:30.989109 1655 webserver.cc:492] Webserver started at https://127.0.0.1:41393/ using document root <none> and password file <none>
I20251214 03:02:30.989274 1655 mini_oidc.cc:186] Starting OIDC Discovery server
I20251214 03:02:30.989502 1655 webserver.cc:492] Webserver started at http://127.0.0.1:40449/ using document root <none> and password file <none>
2025-12-14T03:02:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:30Z Disabled control of system clock
I20251214 03:02:30.997148 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:35159
--webserver_interface=127.1.157.254
--webserver_port=0
--builtin_ntp_servers=127.1.157.212:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:35159
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41393/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/testchainca.cert with env {}
W20251214 03:02:31.103653 7398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:31.103911 7398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:31.103965 7398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:31.107440 7398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:31.107533 7398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:31.107550 7398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:31.107569 7398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:31.107586 7398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251214 03:02:31.107604 7398 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251214 03:02:31.107616 7398 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41393/jwks.json
I20251214 03:02:31.112581 7398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:35159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:35159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41393/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:31.113853 7398 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:31.114734 7398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:31.120251 7403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:31.120332 7406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:31.120288 7404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:31.120501 7398 server_base.cc:1047] running on GCE node
I20251214 03:02:31.121042 7398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:31.122012 7398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:31.123186 7398 hybrid_clock.cc:648] HybridClock initialized: now 1765681351123153 us; error 47 us; skew 500 ppm
I20251214 03:02:31.125128 7398 webserver.cc:492] Webserver started at http://127.1.157.254:38447/ using document root <none> and password file <none>
I20251214 03:02:31.125751 7398 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:31.125815 7398 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:31.125982 7398 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:31.127714 7398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "c6f27124466a48b9830b8bf266feb442"
format_stamp: "Formatted at 2025-12-14 03:02:31 on dist-test-slave-k7z7"
server_key: "3bcac67d2180b671f20e36adea3ae807"
server_key_iv: "2ec25cead97c884a06244bbb494dc631"
server_key_version: "encryptionkey@0"
I20251214 03:02:31.128173 7398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "c6f27124466a48b9830b8bf266feb442"
format_stamp: "Formatted at 2025-12-14 03:02:31 on dist-test-slave-k7z7"
server_key: "3bcac67d2180b671f20e36adea3ae807"
server_key_iv: "2ec25cead97c884a06244bbb494dc631"
server_key_version: "encryptionkey@0"
I20251214 03:02:31.131698 7398 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251214 03:02:31.134002 7412 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:31.134975 7398 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20251214 03:02:31.135085 7398 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "c6f27124466a48b9830b8bf266feb442"
format_stamp: "Formatted at 2025-12-14 03:02:31 on dist-test-slave-k7z7"
server_key: "3bcac67d2180b671f20e36adea3ae807"
server_key_iv: "2ec25cead97c884a06244bbb494dc631"
server_key_version: "encryptionkey@0"
I20251214 03:02:31.135177 7398 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:31.156900 7398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:31.157591 7398 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:31.157745 7398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:31.164536 7398 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:35159
I20251214 03:02:31.164583 7464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:35159 every 8 connection(s)
I20251214 03:02:31.165580 7398 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:31.168267 7465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:31.172214 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 7398
I20251214 03:02:31.172346 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:31.172571 1655 external_mini_cluster.cc:1468] Setting key 11e0ec570baa9c5bd8241c87c010c22d
I20251214 03:02:31.173792 7465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442: Bootstrap starting.
I20251214 03:02:31.175809 7465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:31.176491 7465 log.cc:826] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:31.178258 7465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442: No bootstrap required, opened a new log
I20251214 03:02:31.179613 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:31.173763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.1.157.254:35159)
1214 03:02:31.174341 (+ 578us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:31.174350 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:31.174383 (+ 33us) server_negotiation.cc:408] Connection header received
1214 03:02:31.174890 (+ 507us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:31.174907 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:31.175252 (+ 345us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:31.175532 (+ 280us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:31.175980 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:31.176745 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:31.177412 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:31.177687 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:31.178153 (+ 466us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:31.178171 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:31.178180 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:31.178577 (+ 397us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:31.178618 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:31.178633 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:31.178723 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:31.178863 (+ 140us) server_negotiation.cc:300] Negotiation successful
1214 03:02:31.179065 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":127,"threads_started":1}
I20251214 03:02:31.180742 7465 raft_consensus.cc:359] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f27124466a48b9830b8bf266feb442" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35159 } }
I20251214 03:02:31.180948 7465 raft_consensus.cc:385] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:31.181005 7465 raft_consensus.cc:740] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f27124466a48b9830b8bf266feb442, State: Initialized, Role: FOLLOWER
I20251214 03:02:31.181427 7465 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f27124466a48b9830b8bf266feb442" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35159 } }
I20251214 03:02:31.181567 7465 raft_consensus.cc:399] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:31.181638 7465 raft_consensus.cc:493] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:31.181727 7465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:31.182148 1655 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251214 03:02:31.182613 7465 raft_consensus.cc:515] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f27124466a48b9830b8bf266feb442" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35159 } }
I20251214 03:02:31.182927 7465 leader_election.cc:304] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6f27124466a48b9830b8bf266feb442; no voters:
I20251214 03:02:31.183234 7465 leader_election.cc:290] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:31.183327 7470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:31.183537 7470 raft_consensus.cc:697] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [term 1 LEADER]: Becoming Leader. State: Replica: c6f27124466a48b9830b8bf266feb442, State: Running, Role: LEADER
I20251214 03:02:31.183806 7470 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f27124466a48b9830b8bf266feb442" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35159 } }
I20251214 03:02:31.184273 7465 sys_catalog.cc:565] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:31.185527 7471 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6f27124466a48b9830b8bf266feb442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f27124466a48b9830b8bf266feb442" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35159 } } }
I20251214 03:02:31.185633 7471 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:31.185853 7472 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f27124466a48b9830b8bf266feb442. Latest consensus state: current_term: 1 leader_uuid: "c6f27124466a48b9830b8bf266feb442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f27124466a48b9830b8bf266feb442" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 35159 } } }
I20251214 03:02:31.185932 7472 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442 [sys.catalog]: This master's current role is: LEADER
W20251214 03:02:31.188609 7485 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251214 03:02:31.188670 7485 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251214 03:02:31.188766 7480 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:31.189519 7480 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:31.194556 7480 catalog_manager.cc:1357] Generated new cluster ID: b29d8d41682b47d49cabd5a5b15414ce
I20251214 03:02:31.194617 7480 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:31.217713 7480 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:31.218647 7480 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:31.225216 7480 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c6f27124466a48b9830b8bf266feb442: Generated new TSK 0
I20251214 03:02:31.226013 7480 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 03:02:31.273036 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:31.241889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.1.157.254:35159)
1214 03:02:31.242022 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:31.242026 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:31.242168 (+ 142us) server_negotiation.cc:408] Connection header received
1214 03:02:31.242383 (+ 215us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:31.242386 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:31.242450 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:31.242533 (+ 83us) server_negotiation.cc:227] Negotiated authn=JWT
1214 03:02:31.242997 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:31.243508 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:31.244451 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:31.244620 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:31.244719 (+ 99us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1214 03:02:31.272561 (+ 27842us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1214 03:02:31.272657 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:31.272840 (+ 183us) server_negotiation.cc:300] Negotiation successful
1214 03:02:31.272894 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54,"thread_start_us":126,"threads_started":1}
I20251214 03:02:31.282919 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:31.279450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.1.157.254:35159)
1214 03:02:31.279571 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:31.279575 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:31.279672 (+ 97us) server_negotiation.cc:408] Connection header received
1214 03:02:31.279793 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:31.279795 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:31.279835 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:31.279895 (+ 60us) server_negotiation.cc:227] Negotiated authn=JWT
1214 03:02:31.280293 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:31.280787 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:31.281674 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:31.281818 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:31.281858 (+ 40us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1214 03:02:31.282655 (+ 797us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1214 03:02:31.282826 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60336: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":50}
W20251214 03:02:31.282987 7468 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60336: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251214 03:02:31.283776 7501 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:31.279321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:35159 (local address 127.0.0.1:60336)
1214 03:02:31.279592 (+ 271us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:31.279608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:31.279715 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:31.279917 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:31.279920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:31.279928 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1214 03:02:31.280201 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:31.280208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:31.280893 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:31.280896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:31.281577 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:31.281587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:31.281696 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:31.281719 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1214 03:02:31.282824 (+ 1105us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1214 03:02:31.282985 (+ 161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.157.254:35159: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20251214 03:02:46.290944 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:46.287889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.1.157.254:35159)
1214 03:02:46.288188 (+ 299us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:46.288192 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:46.288208 (+ 16us) server_negotiation.cc:408] Connection header received
1214 03:02:46.288260 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:46.288263 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:46.288313 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:46.288391 (+ 78us) server_negotiation.cc:227] Negotiated authn=JWT
1214 03:02:46.288707 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.289286 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:46.289868 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.290008 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:46.290051 (+ 43us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1214 03:02:46.290686 (+ 635us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1214 03:02:46.290829 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40662: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
W20251214 03:02:46.291007 7507 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40662: JWT verification failed: token expired
W20251214 03:02:46.290999 7506 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:46.287747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:35159 (local address 127.0.0.1:40662)
1214 03:02:46.288025 (+ 278us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:46.288042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:46.288150 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:46.288407 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:46.288409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:46.288415 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
1214 03:02:46.288617 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:46.288623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.289368 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:46.289370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:46.289767 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:46.289774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.289858 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:46.289877 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1214 03:02:46.290775 (+ 898us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1214 03:02:46.290886 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.157.254:35159: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
W20251214 03:02:46.293512 7507 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:46.292661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.1.157.254:35159)
1214 03:02:46.292792 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:46.292797 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:46.292862 (+ 65us) server_negotiation.cc:408] Connection header received
1214 03:02:46.292993 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:46.292997 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:46.293045 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:46.293106 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:46.293403 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40676: BlockingRecv error: recv got EOF from 127.0.0.1:40676 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20251214 03:02:46.296787 7517 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.1.157.254:35159
W20251214 03:02:46.297345 7507 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:46.296574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.1.157.254:35159)
1214 03:02:46.296687 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:46.296690 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:46.296729 (+ 39us) server_negotiation.cc:408] Connection header received
1214 03:02:46.296950 (+ 221us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:46.296954 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:46.296999 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:46.297064 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:46.297243 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40690: BlockingRecv error: recv got EOF from 127.0.0.1:40690 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20251214 03:02:46.297561 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 7398
2025-12-14T03:02:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15349 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251214 03:02:46.337061 1655 mini_oidc.cc:150] Starting JWKS server
I20251214 03:02:46.338039 1655 webserver.cc:492] Webserver started at https://127.0.0.1:39157/ using document root <none> and password file <none>
I20251214 03:02:46.338124 1655 mini_oidc.cc:186] Starting OIDC Discovery server
I20251214 03:02:46.338313 1655 webserver.cc:492] Webserver started at http://127.0.0.1:35857/ using document root <none> and password file <none>
2025-12-14T03:02:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:46Z Disabled control of system clock
I20251214 03:02:46.346120 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:34457
--webserver_interface=127.1.157.254
--webserver_port=0
--builtin_ntp_servers=127.1.157.212:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:34457
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39157/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/testchainca.cert with env {}
W20251214 03:02:46.452044 7523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:46.452287 7523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:46.452337 7523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:46.455555 7523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:46.455610 7523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:46.455624 7523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:46.455641 7523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:46.455657 7523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251214 03:02:46.455672 7523 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251214 03:02:46.455693 7523 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39157/jwks.json
I20251214 03:02:46.460193 7523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:34457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:34457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39157/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:46.461244 7523 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:46.462033 7523 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:46.467907 7529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:46.468037 7523 server_base.cc:1047] running on GCE node
W20251214 03:02:46.468116 7531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:46.468037 7528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:46.468662 7523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:46.469646 7523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:46.470810 7523 hybrid_clock.cc:648] HybridClock initialized: now 1765681366470799 us; error 33 us; skew 500 ppm
I20251214 03:02:46.472636 7523 webserver.cc:492] Webserver started at http://127.1.157.254:45501/ using document root <none> and password file <none>
I20251214 03:02:46.473223 7523 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:46.473295 7523 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:46.473488 7523 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:46.475147 7523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "45dd668f92ab4a378a04b9c3a3872827"
format_stamp: "Formatted at 2025-12-14 03:02:46 on dist-test-slave-k7z7"
server_key: "f9e4f0766cf914b0c734f134430897c4"
server_key_iv: "39e44e1178bc2f9b4d4e174c06b2f2b4"
server_key_version: "encryptionkey@0"
I20251214 03:02:46.475613 7523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "45dd668f92ab4a378a04b9c3a3872827"
format_stamp: "Formatted at 2025-12-14 03:02:46 on dist-test-slave-k7z7"
server_key: "f9e4f0766cf914b0c734f134430897c4"
server_key_iv: "39e44e1178bc2f9b4d4e174c06b2f2b4"
server_key_version: "encryptionkey@0"
I20251214 03:02:46.479041 7523 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251214 03:02:46.481416 7537 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:46.482546 7523 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251214 03:02:46.482677 7523 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "45dd668f92ab4a378a04b9c3a3872827"
format_stamp: "Formatted at 2025-12-14 03:02:46 on dist-test-slave-k7z7"
server_key: "f9e4f0766cf914b0c734f134430897c4"
server_key_iv: "39e44e1178bc2f9b4d4e174c06b2f2b4"
server_key_version: "encryptionkey@0"
I20251214 03:02:46.482780 7523 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:46.494200 7523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:46.494861 7523 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:46.495033 7523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:46.502496 7523 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:34457
I20251214 03:02:46.502498 7589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:34457 every 8 connection(s)
I20251214 03:02:46.503543 7523 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:46.506410 7590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:46.511252 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 7523
I20251214 03:02:46.511372 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:46.511619 1655 external_mini_cluster.cc:1468] Setting key d3ceda5c46d33e9aed1edb1e6922bdee
I20251214 03:02:46.512136 7590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827: Bootstrap starting.
I20251214 03:02:46.514477 7590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:46.515362 7590 log.cc:826] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:46.517535 7590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827: No bootstrap required, opened a new log
I20251214 03:02:46.520061 7590 raft_consensus.cc:359] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dd668f92ab4a378a04b9c3a3872827" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 34457 } }
I20251214 03:02:46.520272 7590 raft_consensus.cc:385] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:46.520331 7590 raft_consensus.cc:740] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dd668f92ab4a378a04b9c3a3872827, State: Initialized, Role: FOLLOWER
I20251214 03:02:46.520745 7590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dd668f92ab4a378a04b9c3a3872827" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 34457 } }
I20251214 03:02:46.520869 7590 raft_consensus.cc:399] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:46.520932 7590 raft_consensus.cc:493] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:46.521023 7590 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:46.521922 7590 raft_consensus.cc:515] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dd668f92ab4a378a04b9c3a3872827" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 34457 } }
I20251214 03:02:46.522271 7590 leader_election.cc:304] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45dd668f92ab4a378a04b9c3a3872827; no voters:
I20251214 03:02:46.522557 7590 leader_election.cc:290] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:46.522648 7595 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:46.522835 7595 raft_consensus.cc:697] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [term 1 LEADER]: Becoming Leader. State: Replica: 45dd668f92ab4a378a04b9c3a3872827, State: Running, Role: LEADER
I20251214 03:02:46.523177 7595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dd668f92ab4a378a04b9c3a3872827" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 34457 } }
I20251214 03:02:46.523746 7590 sys_catalog.cc:565] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:46.524471 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:46.512840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.1.157.254:34457)
1214 03:02:46.513493 (+ 653us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:46.513503 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:46.513540 (+ 37us) server_negotiation.cc:408] Connection header received
1214 03:02:46.514099 (+ 559us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:46.514117 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:46.514494 (+ 377us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:46.514806 (+ 312us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:46.515339 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.516102 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:46.516978 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.517228 (+ 250us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:46.522368 (+ 5140us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:46.522394 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:46.522410 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:46.522922 (+ 512us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:46.522990 (+ 68us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:46.523013 (+ 23us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:46.523509 (+ 496us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:46.523600 (+ 91us) server_negotiation.cc:300] Negotiation successful
1214 03:02:46.523811 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20251214 03:02:46.524685 7596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45dd668f92ab4a378a04b9c3a3872827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dd668f92ab4a378a04b9c3a3872827" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 34457 } } }
I20251214 03:02:46.524801 7596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:46.525233 7604 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:46.525297 7597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45dd668f92ab4a378a04b9c3a3872827. Latest consensus state: current_term: 1 leader_uuid: "45dd668f92ab4a378a04b9c3a3872827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dd668f92ab4a378a04b9c3a3872827" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 34457 } } }
I20251214 03:02:46.525380 7597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:46.528360 1655 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251214 03:02:46.529388 7604 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:46.534435 7604 catalog_manager.cc:1357] Generated new cluster ID: 8c966a2899a947d5b458a2626101af55
I20251214 03:02:46.534514 7604 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:46.556840 7604 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:46.558125 7604 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:46.571286 7604 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 45dd668f92ab4a378a04b9c3a3872827: Generated new TSK 0
I20251214 03:02:46.572041 7604 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251214 03:02:46.598336 7620 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251214 03:02:46.598438 7620 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251214 03:02:46.599161 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:46.582907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.1.157.254:34457)
1214 03:02:46.583038 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:46.583043 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:46.583172 (+ 129us) server_negotiation.cc:408] Connection header received
1214 03:02:46.583335 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:46.583339 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:46.583406 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:46.583523 (+ 117us) server_negotiation.cc:227] Negotiated authn=JWT
1214 03:02:46.583861 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.584691 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:46.585376 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.585643 (+ 267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:46.585766 (+ 123us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1214 03:02:46.598731 (+ 12965us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1214 03:02:46.599016 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20251214 03:02:46.599234 7619 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:46.582786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:34457 (local address 127.0.0.1:36922)
1214 03:02:46.583102 (+ 316us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:46.583114 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:46.583203 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:46.583526 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:46.583529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:46.583536 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
1214 03:02:46.583760 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:46.583767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.584807 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:46.584810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:46.585280 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:46.585288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.585376 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:46.585397 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1214 03:02:46.598936 (+ 13539us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1214 03:02:46.599074 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.157.254:34457: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
W20251214 03:02:46.599243 7593 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20251214 03:02:46.599857 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 7523
2025-12-14T03:02:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (301 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251214 03:02:46.638300 1655 mini_oidc.cc:150] Starting JWKS server
I20251214 03:02:46.639149 1655 webserver.cc:492] Webserver started at https://127.0.0.1:44425/ using document root <none> and password file <none>
I20251214 03:02:46.639240 1655 mini_oidc.cc:186] Starting OIDC Discovery server
I20251214 03:02:46.639442 1655 webserver.cc:492] Webserver started at http://127.0.0.1:36099/ using document root <none> and password file <none>
2025-12-14T03:02:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:46Z Disabled control of system clock
I20251214 03:02:46.646473 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:46685
--webserver_interface=127.1.157.254
--webserver_port=0
--builtin_ntp_servers=127.1.157.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:46685
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44425/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251214 03:02:46.750875 7626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:46.751137 7626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:46.751186 7626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:46.754542 7626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:46.754613 7626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:46.754628 7626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:46.754647 7626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:46.754663 7626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251214 03:02:46.754679 7626 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251214 03:02:46.754693 7626 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44425/jwks.json
I20251214 03:02:46.758689 7626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:46685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:46685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44425/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:46.759737 7626 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:46.760777 7626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:46.766131 7631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:46.766132 7634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:46.766182 7632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:46.766215 7626 server_base.cc:1047] running on GCE node
I20251214 03:02:46.766896 7626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:46.767786 7626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:46.768978 7626 hybrid_clock.cc:648] HybridClock initialized: now 1765681366768955 us; error 36 us; skew 500 ppm
I20251214 03:02:46.770869 7626 webserver.cc:492] Webserver started at http://127.1.157.254:36327/ using document root <none> and password file <none>
I20251214 03:02:46.771409 7626 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:46.771461 7626 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:46.771627 7626 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:46.773241 7626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "5285691eafeb422687a7e64a8bdf7f80"
format_stamp: "Formatted at 2025-12-14 03:02:46 on dist-test-slave-k7z7"
server_key: "aebbb108af1b39adbe8f5f79d68d99b7"
server_key_iv: "655a4a526eda63ebe25d326146d50352"
server_key_version: "encryptionkey@0"
I20251214 03:02:46.773681 7626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "5285691eafeb422687a7e64a8bdf7f80"
format_stamp: "Formatted at 2025-12-14 03:02:46 on dist-test-slave-k7z7"
server_key: "aebbb108af1b39adbe8f5f79d68d99b7"
server_key_iv: "655a4a526eda63ebe25d326146d50352"
server_key_version: "encryptionkey@0"
I20251214 03:02:46.776770 7626 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:02:46.779011 7640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:46.780090 7626 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251214 03:02:46.780194 7626 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "5285691eafeb422687a7e64a8bdf7f80"
format_stamp: "Formatted at 2025-12-14 03:02:46 on dist-test-slave-k7z7"
server_key: "aebbb108af1b39adbe8f5f79d68d99b7"
server_key_iv: "655a4a526eda63ebe25d326146d50352"
server_key_version: "encryptionkey@0"
I20251214 03:02:46.780287 7626 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:46.807533 7626 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:46.808190 7626 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:46.808358 7626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:46.815383 7692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:46685 every 8 connection(s)
I20251214 03:02:46.815387 7626 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:46685
I20251214 03:02:46.816424 7626 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:46.819156 7693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:46.821475 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 7626
I20251214 03:02:46.821741 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:46.822003 1655 external_mini_cluster.cc:1468] Setting key 84919b228531138794a57553fca7b39d
I20251214 03:02:46.825500 7693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80: Bootstrap starting.
I20251214 03:02:46.827708 7693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:46.828577 7693 log.cc:826] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:46.829665 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:46.823284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.1.157.254:46685)
1214 03:02:46.823917 (+ 633us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:46.823927 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:46.823961 (+ 34us) server_negotiation.cc:408] Connection header received
1214 03:02:46.824526 (+ 565us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:46.824546 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:46.824895 (+ 349us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:46.825224 (+ 329us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:46.825714 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.826451 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:46.827106 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.827376 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:46.827887 (+ 511us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:46.827909 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:46.827924 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:46.828382 (+ 458us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:46.828447 (+ 65us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:46.828468 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:46.828575 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:46.828729 (+ 154us) server_negotiation.cc:300] Negotiation successful
1214 03:02:46.828963 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
I20251214 03:02:46.830669 7693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80: No bootstrap required, opened a new log
I20251214 03:02:46.832509 1655 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251214 03:02:46.833482 7693 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5285691eafeb422687a7e64a8bdf7f80" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46685 } }
I20251214 03:02:46.833770 7693 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:46.833846 7693 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5285691eafeb422687a7e64a8bdf7f80, State: Initialized, Role: FOLLOWER
I20251214 03:02:46.834303 7693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5285691eafeb422687a7e64a8bdf7f80" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46685 } }
I20251214 03:02:46.834439 7693 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:46.834502 7693 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:46.834594 7693 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:46.835489 7693 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5285691eafeb422687a7e64a8bdf7f80" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46685 } }
I20251214 03:02:46.835856 7693 leader_election.cc:304] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5285691eafeb422687a7e64a8bdf7f80; no voters:
I20251214 03:02:46.836169 7693 leader_election.cc:290] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:46.836413 7698 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:46.836618 7698 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [term 1 LEADER]: Becoming Leader. State: Replica: 5285691eafeb422687a7e64a8bdf7f80, State: Running, Role: LEADER
I20251214 03:02:46.836908 7698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5285691eafeb422687a7e64a8bdf7f80" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46685 } }
I20251214 03:02:46.837422 7693 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:46.838534 7699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5285691eafeb422687a7e64a8bdf7f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5285691eafeb422687a7e64a8bdf7f80" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46685 } } }
I20251214 03:02:46.838671 7699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:46.838538 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5285691eafeb422687a7e64a8bdf7f80. Latest consensus state: current_term: 1 leader_uuid: "5285691eafeb422687a7e64a8bdf7f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5285691eafeb422687a7e64a8bdf7f80" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46685 } } }
I20251214 03:02:46.839005 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80 [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:46.839007 7707 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:46.842697 7707 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:46.848050 7707 catalog_manager.cc:1357] Generated new cluster ID: eb6b07d962034204b4eed53c9d88e910
I20251214 03:02:46.848119 7707 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:46.858405 7707 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:46.859226 7707 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:46.870903 7707 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5285691eafeb422687a7e64a8bdf7f80: Generated new TSK 0
I20251214 03:02:46.871668 7707 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251214 03:02:46.910022 7723 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251214 03:02:46.910115 7723 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20251214 03:02:46.911840 7722 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:46.886799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:46685 (local address 127.0.0.1:51578)
1214 03:02:46.887104 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:46.887118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:46.887278 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:46.887582 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:46.887585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:46.887593 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1214 03:02:46.887874 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:46.887884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.888591 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:46.888595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:46.889281 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:46.889292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.889411 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:46.889442 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1214 03:02:46.911454 (+ 22012us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1214 03:02:46.911631 (+ 177us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.157.254:46685: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":112,"threads_started":1}
I20251214 03:02:46.912655 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 7626
I20251214 03:02:46.912684 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:46.886921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.1.157.254:46685)
1214 03:02:46.887049 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:46.887053 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:46.887188 (+ 135us) server_negotiation.cc:408] Connection header received
1214 03:02:46.887413 (+ 225us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:46.887416 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:46.887471 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:46.887567 (+ 96us) server_negotiation.cc:227] Negotiated authn=JWT
1214 03:02:46.887980 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.888481 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:46.889414 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:46.889582 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:46.889671 (+ 89us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1214 03:02:46.911109 (+ 21438us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1214 03:02:46.912530 (+ 1421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51578: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
2025-12-14T03:02:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-12-14T03:02:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:46Z Disabled control of system clock
I20251214 03:02:46.960600 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:43139
--webserver_interface=127.1.157.254
--webserver_port=0
--builtin_ntp_servers=127.1.157.212:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:43139
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/testchainca.cert with env {}
W20251214 03:02:47.074167 7727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:47.074417 7727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:47.074468 7727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:47.077795 7727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:47.077858 7727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:47.077873 7727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:47.077890 7727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:47.077906 7727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251214 03:02:47.077924 7727 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251214 03:02:47.077939 7727 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251214 03:02:47.082432 7727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:43139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:43139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:47.083491 7727 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:47.084270 7727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:47.089908 7735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:47.089915 7733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:47.089929 7732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:47.090001 7727 server_base.cc:1047] running on GCE node
I20251214 03:02:47.090555 7727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:47.091442 7727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:47.092612 7727 hybrid_clock.cc:648] HybridClock initialized: now 1765681367092588 us; error 38 us; skew 500 ppm
I20251214 03:02:47.094300 7727 webserver.cc:492] Webserver started at http://127.1.157.254:39489/ using document root <none> and password file <none>
I20251214 03:02:47.094821 7727 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:47.094871 7727 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:47.095014 7727 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:47.096657 7727 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "51b1a4142dd94023b555e68afd504bda"
format_stamp: "Formatted at 2025-12-14 03:02:47 on dist-test-slave-k7z7"
server_key: "d423a02a987307c9d37b8d8c46354dba"
server_key_iv: "657815897a1774ca5fb1117cb9746035"
server_key_version: "encryptionkey@0"
I20251214 03:02:47.097091 7727 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "51b1a4142dd94023b555e68afd504bda"
format_stamp: "Formatted at 2025-12-14 03:02:47 on dist-test-slave-k7z7"
server_key: "d423a02a987307c9d37b8d8c46354dba"
server_key_iv: "657815897a1774ca5fb1117cb9746035"
server_key_version: "encryptionkey@0"
I20251214 03:02:47.100344 7727 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251214 03:02:47.102591 7741 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:47.103636 7727 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:47.103737 7727 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "51b1a4142dd94023b555e68afd504bda"
format_stamp: "Formatted at 2025-12-14 03:02:47 on dist-test-slave-k7z7"
server_key: "d423a02a987307c9d37b8d8c46354dba"
server_key_iv: "657815897a1774ca5fb1117cb9746035"
server_key_version: "encryptionkey@0"
I20251214 03:02:47.103822 7727 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:47.115015 7727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:47.115686 7727 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:47.115837 7727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:47.123261 7727 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:43139
I20251214 03:02:47.123311 7793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:43139 every 8 connection(s)
I20251214 03:02:47.124331 7727 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:47.125761 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 7727
I20251214 03:02:47.125892 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:47.126138 1655 external_mini_cluster.cc:1468] Setting key fe098a00b2592de3f951a7a66c1f6790
I20251214 03:02:47.127938 7794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:47.134086 7794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda: Bootstrap starting.
I20251214 03:02:47.134346 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:47.127557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.1.157.254:43139)
1214 03:02:47.128270 (+ 713us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:47.128279 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:47.128317 (+ 38us) server_negotiation.cc:408] Connection header received
1214 03:02:47.128967 (+ 650us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:47.128984 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:47.129402 (+ 418us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:47.129703 (+ 301us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:47.130253 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:47.131010 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:47.131875 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:47.132092 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:47.132695 (+ 603us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:47.132741 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:47.132752 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:02:47.133134 (+ 382us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:47.133211 (+ 77us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:02:47.133257 (+ 46us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:47.133341 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:47.133488 (+ 147us) server_negotiation.cc:300] Negotiation successful
1214 03:02:47.133726 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":125,"threads_started":1}
I20251214 03:02:47.136219 7794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:47.136813 1655 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251214 03:02:47.136925 7794 log.cc:826] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:47.138849 7794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda: No bootstrap required, opened a new log
I20251214 03:02:47.141326 7794 raft_consensus.cc:359] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1a4142dd94023b555e68afd504bda" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43139 } }
I20251214 03:02:47.141520 7794 raft_consensus.cc:385] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:47.141577 7794 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b1a4142dd94023b555e68afd504bda, State: Initialized, Role: FOLLOWER
I20251214 03:02:47.141971 7794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1a4142dd94023b555e68afd504bda" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43139 } }
I20251214 03:02:47.142104 7794 raft_consensus.cc:399] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:47.142161 7794 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:47.142258 7794 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:47.143064 7794 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1a4142dd94023b555e68afd504bda" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43139 } }
I20251214 03:02:47.143406 7794 leader_election.cc:304] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51b1a4142dd94023b555e68afd504bda; no voters:
I20251214 03:02:47.143677 7794 leader_election.cc:290] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:47.143819 7799 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:47.144011 7799 raft_consensus.cc:697] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [term 1 LEADER]: Becoming Leader. State: Replica: 51b1a4142dd94023b555e68afd504bda, State: Running, Role: LEADER
I20251214 03:02:47.144304 7799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1a4142dd94023b555e68afd504bda" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43139 } }
I20251214 03:02:47.144814 7794 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:47.145967 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51b1a4142dd94023b555e68afd504bda. Latest consensus state: current_term: 1 leader_uuid: "51b1a4142dd94023b555e68afd504bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1a4142dd94023b555e68afd504bda" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43139 } } }
I20251214 03:02:47.146088 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:47.146163 7800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51b1a4142dd94023b555e68afd504bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1a4142dd94023b555e68afd504bda" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 43139 } } }
I20251214 03:02:47.146246 7800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:47.146668 7808 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:47.149240 7808 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:47.155967 7808 catalog_manager.cc:1357] Generated new cluster ID: 6a03cb9b3fee4110b0cb8dc1283938aa
I20251214 03:02:47.156069 7808 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:47.165933 7808 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:47.166913 7808 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:47.173041 7808 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 51b1a4142dd94023b555e68afd504bda: Generated new TSK 0
I20251214 03:02:47.173763 7808 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 03:02:47.220567 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:47.186684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.1.157.254:43139)
1214 03:02:47.186807 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:47.186812 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:47.186943 (+ 131us) server_negotiation.cc:408] Connection header received
1214 03:02:47.187113 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:47.187116 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:47.187182 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:47.187271 (+ 89us) server_negotiation.cc:227] Negotiated authn=JWT
1214 03:02:47.187622 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:47.188187 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:47.188840 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:47.189018 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:47.189109 (+ 91us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1214 03:02:47.220079 (+ 30970us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1214 03:02:47.220393 (+ 314us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46052: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20251214 03:02:47.220629 7823 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:47.186609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:43139 (local address 127.0.0.1:46052)
1214 03:02:47.186870 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:47.186882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:47.186981 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:47.187288 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:47.187292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:47.187300 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1214 03:02:47.187519 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:47.187525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:47.188310 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:47.188314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:47.188735 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:47.188743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:47.188850 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:47.188873 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1214 03:02:47.220296 (+ 31423us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1214 03:02:47.220453 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.157.254:43139: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
W20251214 03:02:47.220657 7797 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46052: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
I20251214 03:02:47.221107 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 7727
2025-12-14T03:02:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (285 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (111 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (118 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:02:47 dist-test-slave-k7z7 krb5kdc[7835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:02:47 dist-test-slave-k7z7 krb5kdc[7835](info): set up 2 sockets
Dec 14 03:02:47 dist-test-slave-k7z7 krb5kdc[7835](info): commencing operation
krb5kdc: starting...
W20251214 03:02:49.293464 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.824s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:02:49 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:02:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:49.424970 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:46519
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:46519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:49.533768 7851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:49.534004 7851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:49.534080 7851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:49.537339 7851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:49.537403 7851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:49.537434 7851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:49.537467 7851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:49.537498 7851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:49.541739 7851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:46519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:49.542768 7851 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:49.543573 7851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251214 03:02:49.548969 7851 server_base.cc:1047] running on GCE node
W20251214 03:02:49.548911 7857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:49.548911 7859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:49.549129 7856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:49.549633 7851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:49.550535 7851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:49.551704 7851 hybrid_clock.cc:648] HybridClock initialized: now 1765681369551688 us; error 34 us; skew 500 ppm
Dec 14 03:02:49 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681369, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:49.554175 7851 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:02:49.555174 7851 webserver.cc:492] Webserver started at http://127.1.157.254:39151/ using document root <none> and password file <none>
I20251214 03:02:49.555701 7851 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:49.555749 7851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:49.555915 7851 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:49.557665 7851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "4b8c8707a7e64c9e930804cd51ec1a5e"
format_stamp: "Formatted at 2025-12-14 03:02:49 on dist-test-slave-k7z7"
server_key: "274acb83c0173bd3a5acbb86430a9bef"
server_key_iv: "fac1cd7d5fa6f16560f0d7f5578a1213"
server_key_version: "encryptionkey@0"
I20251214 03:02:49.558106 7851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "4b8c8707a7e64c9e930804cd51ec1a5e"
format_stamp: "Formatted at 2025-12-14 03:02:49 on dist-test-slave-k7z7"
server_key: "274acb83c0173bd3a5acbb86430a9bef"
server_key_iv: "fac1cd7d5fa6f16560f0d7f5578a1213"
server_key_version: "encryptionkey@0"
I20251214 03:02:49.561332 7851 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251214 03:02:49.563454 7866 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:49.564462 7851 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:49.564568 7851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "4b8c8707a7e64c9e930804cd51ec1a5e"
format_stamp: "Formatted at 2025-12-14 03:02:49 on dist-test-slave-k7z7"
server_key: "274acb83c0173bd3a5acbb86430a9bef"
server_key_iv: "fac1cd7d5fa6f16560f0d7f5578a1213"
server_key_version: "encryptionkey@0"
I20251214 03:02:49.564656 7851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:49.604215 7851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:49.607012 7851 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:49.607189 7851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:49.614022 7851 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:46519
I20251214 03:02:49.614022 7918 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:46519 every 8 connection(s)
I20251214 03:02:49.615023 7851 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:49.617666 7919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:49.620875 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 7851
I20251214 03:02:49.620987 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:49.621232 1655 external_mini_cluster.cc:1468] Setting key 0d60e1a9ea3d11f98f8691ac6920b1c5
I20251214 03:02:49.623150 7919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e: Bootstrap starting.
I20251214 03:02:49.625190 7919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:49.625898 7919 log.cc:826] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:49.627804 7919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e: No bootstrap required, opened a new log
Dec 14 03:02:49 dist-test-slave-k7z7 krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:49.630801 7919 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8c8707a7e64c9e930804cd51ec1a5e" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46519 } }
I20251214 03:02:49.630982 7919 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:49.631055 7919 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8c8707a7e64c9e930804cd51ec1a5e, State: Initialized, Role: FOLLOWER
I20251214 03:02:49.631507 7919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8c8707a7e64c9e930804cd51ec1a5e" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46519 } }
I20251214 03:02:49.631652 7919 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:49.631713 7919 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:49.631799 7919 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:49.632800 7919 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8c8707a7e64c9e930804cd51ec1a5e" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46519 } }
I20251214 03:02:49.633154 7919 leader_election.cc:304] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b8c8707a7e64c9e930804cd51ec1a5e; no voters:
I20251214 03:02:49.633518 7919 leader_election.cc:290] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:49.633579 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:49.622438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.1.157.254:46519)
1214 03:02:49.622842 (+ 404us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:49.622852 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:49.622884 (+ 32us) server_negotiation.cc:408] Connection header received
1214 03:02:49.623419 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:49.623438 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:49.623725 (+ 287us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:49.624011 (+ 286us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:49.624879 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:49.625678 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:49.626335 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:49.626606 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:49.628561 (+ 1955us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:49.628580 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:49.628593 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:49.628632 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:49.630638 (+ 2006us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:49.631041 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:49.631047 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:49.631052 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:49.631144 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:49.631387 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:49.631391 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:49.631392 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:49.631792 (+ 400us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:49.632003 (+ 211us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:49.632185 (+ 182us) server_negotiation.cc:300] Negotiation successful
1214 03:02:49.632580 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
I20251214 03:02:49.633621 7924 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:49.633908 7924 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [term 1 LEADER]: Becoming Leader. State: Replica: 4b8c8707a7e64c9e930804cd51ec1a5e, State: Running, Role: LEADER
I20251214 03:02:49.634186 7924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8c8707a7e64c9e930804cd51ec1a5e" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46519 } }
I20251214 03:02:49.634604 7919 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:49.635270 7924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b8c8707a7e64c9e930804cd51ec1a5e. Latest consensus state: current_term: 1 leader_uuid: "4b8c8707a7e64c9e930804cd51ec1a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8c8707a7e64c9e930804cd51ec1a5e" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46519 } } }
I20251214 03:02:49.635965 7924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:49.635846 7925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b8c8707a7e64c9e930804cd51ec1a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8c8707a7e64c9e930804cd51ec1a5e" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46519 } } }
I20251214 03:02:49.636219 7925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:49.636433 7935 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:49.638819 7935 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:49.643477 7935 catalog_manager.cc:1357] Generated new cluster ID: a3dc4859175247098115da3d5b752cd4
I20251214 03:02:49.643536 7935 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:49.656889 7935 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:49.657919 7935 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:49.662279 7935 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4b8c8707a7e64c9e930804cd51ec1a5e: Generated new TSK 0
I20251214 03:02:49.662883 7935 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:49.708866 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:46519
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:49.815621 7947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:49.815825 7947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:49.815905 7947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:49.819187 7947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:49.819248 7947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:49.819346 7947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:49.823401 7947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:49.824429 7947 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:49.825297 7947 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:49.831705 7952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:49.831720 7953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:49.831869 7947 server_base.cc:1047] running on GCE node
W20251214 03:02:49.831777 7955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:49.832329 7947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:49.832904 7947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:49.834077 7947 hybrid_clock.cc:648] HybridClock initialized: now 1765681369834051 us; error 40 us; skew 500 ppm
Dec 14 03:02:49 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681369, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:49.836974 7947 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:02:49.838135 7947 webserver.cc:492] Webserver started at http://127.1.157.193:38635/ using document root <none> and password file <none>
I20251214 03:02:49.838685 7947 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:49.838734 7947 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:49.838905 7947 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:49.840584 7947 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "812bf09bd6fc4fe0aca8d42cf3208b65"
format_stamp: "Formatted at 2025-12-14 03:02:49 on dist-test-slave-k7z7"
server_key: "70e3223f6ff960dc54debda19d35df57"
server_key_iv: "1ce6d15d35f66aa545d4fa71f1426d63"
server_key_version: "encryptionkey@0"
I20251214 03:02:49.841024 7947 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "812bf09bd6fc4fe0aca8d42cf3208b65"
format_stamp: "Formatted at 2025-12-14 03:02:49 on dist-test-slave-k7z7"
server_key: "70e3223f6ff960dc54debda19d35df57"
server_key_iv: "1ce6d15d35f66aa545d4fa71f1426d63"
server_key_version: "encryptionkey@0"
I20251214 03:02:49.844276 7947 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251214 03:02:49.846434 7962 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:49.847429 7947 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:49.847523 7947 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "812bf09bd6fc4fe0aca8d42cf3208b65"
format_stamp: "Formatted at 2025-12-14 03:02:49 on dist-test-slave-k7z7"
server_key: "70e3223f6ff960dc54debda19d35df57"
server_key_iv: "1ce6d15d35f66aa545d4fa71f1426d63"
server_key_version: "encryptionkey@0"
I20251214 03:02:49.847611 7947 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:49.857702 7947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:49.860313 7947 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:49.860471 7947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:49.860982 7947 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:49.861845 7947 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:49.861892 7947 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:49.861930 7947 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:49.861956 7947 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:49.870550 7947 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:34163
I20251214 03:02:49.870575 8075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:34163 every 8 connection(s)
I20251214 03:02:49.871516 7947 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:02:49.874955 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 7947
I20251214 03:02:49.875052 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:02:49.875293 1655 external_mini_cluster.cc:1468] Setting key 5ac9081545d34af67ef4978bb71ff57d
Dec 14 03:02:49 dist-test-slave-k7z7 krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681369, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:02:49.884286 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:49.873286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:54913 (local address 127.1.157.254:46519)
1214 03:02:49.873425 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:49.873429 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:49.874082 (+ 653us) server_negotiation.cc:408] Connection header received
1214 03:02:49.874937 (+ 855us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:49.874940 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:49.874999 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:49.875072 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:49.876865 (+ 1793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:49.877421 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:49.878159 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:49.878330 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:49.880639 (+ 2309us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:49.880658 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:49.880660 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:49.880685 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:49.882081 (+ 1396us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:49.882693 (+ 612us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:49.882696 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:49.882698 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:49.882751 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:49.883086 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:49.883089 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:49.883091 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:49.883263 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:49.883343 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:49.883987 (+ 644us) server_negotiation.cc:300] Negotiation successful
1214 03:02:49.884119 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251214 03:02:49.884953 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:49.873521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:46519 (local address 127.1.157.193:54913)
1214 03:02:49.873941 (+ 420us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:49.873976 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:49.874744 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:49.875255 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:49.875263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:49.875711 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:49.876724 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:49.876740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:49.877541 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:49.877544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:49.878043 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:49.878053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:49.878278 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:49.878859 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:49.878890 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:49.880472 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:49.882216 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:49.882222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:49.882235 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:49.882568 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:49.882851 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:49.882855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:49.882857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:49.882972 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:49.883373 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:49.883381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:49.883682 (+ 301us) client_negotiation.cc:769] Sending connection context
1214 03:02:49.883918 (+ 236us) client_negotiation.cc:240] Negotiation successful
1214 03:02:49.884218 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20251214 03:02:49.886277 8076 heartbeater.cc:344] Connected to a master server at 127.1.157.254:46519
I20251214 03:02:49.886510 8076 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
I20251214 03:02:49.887079 8076 heartbeater.cc:507] Master 127.1.157.254:46519 requested a full tablet report, sending...
I20251214 03:02:49.888505 7883 ts_manager.cc:194] Registered new tserver with Master: 812bf09bd6fc4fe0aca8d42cf3208b65 (127.1.157.193:34163)
I20251214 03:02:49.890064 7883 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:54913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:49.924779 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:46519
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:50.031672 8083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:50.031876 8083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:50.031934 8083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:50.035195 8083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:50.035315 8083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:50.035410 8083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:50.039572 8083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:50.040526 8083 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:50.041323 8083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:50.047701 8088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:50.047691 8089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:50.047691 8091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:50.047824 8083 server_base.cc:1047] running on GCE node
I20251214 03:02:50.048241 8083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:50.048734 8083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:50.049871 8083 hybrid_clock.cc:648] HybridClock initialized: now 1765681370049862 us; error 22 us; skew 500 ppm
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:50.052433 8083 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:02:50.053397 8083 webserver.cc:492] Webserver started at http://127.1.157.194:36241/ using document root <none> and password file <none>
I20251214 03:02:50.053884 8083 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:50.053925 8083 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:50.054065 8083 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:50.055621 8083 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "5b0d20de34f1494eaeddf2384d47b909"
format_stamp: "Formatted at 2025-12-14 03:02:50 on dist-test-slave-k7z7"
server_key: "d9ca369c406b3ee1a97839a1516135f1"
server_key_iv: "1d9fe01e7230c1f370d44766fa7b97e2"
server_key_version: "encryptionkey@0"
I20251214 03:02:50.056010 8083 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "5b0d20de34f1494eaeddf2384d47b909"
format_stamp: "Formatted at 2025-12-14 03:02:50 on dist-test-slave-k7z7"
server_key: "d9ca369c406b3ee1a97839a1516135f1"
server_key_iv: "1d9fe01e7230c1f370d44766fa7b97e2"
server_key_version: "encryptionkey@0"
I20251214 03:02:50.059070 8083 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:02:50.061094 8098 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:50.062184 8083 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:50.062285 8083 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "5b0d20de34f1494eaeddf2384d47b909"
format_stamp: "Formatted at 2025-12-14 03:02:50 on dist-test-slave-k7z7"
server_key: "d9ca369c406b3ee1a97839a1516135f1"
server_key_iv: "1d9fe01e7230c1f370d44766fa7b97e2"
server_key_version: "encryptionkey@0"
I20251214 03:02:50.062384 8083 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:50.096320 8083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:50.099498 8083 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:50.099666 8083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:50.100172 8083 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:50.101080 8083 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:50.101128 8083 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:50.101166 8083 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:50.101220 8083 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:50.109812 8083 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:37463
I20251214 03:02:50.109849 8211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:37463 every 8 connection(s)
I20251214 03:02:50.110773 8083 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:50.120774 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 8083
I20251214 03:02:50.120870 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:02:50.121088 1655 external_mini_cluster.cc:1468] Setting key f3e01cb66a4114cb8352138b7b4b1fdb
I20251214 03:02:50.122556 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:50.112504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:40317 (local address 127.1.157.254:46519)
1214 03:02:50.112673 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:50.112678 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:50.113305 (+ 627us) server_negotiation.cc:408] Connection header received
1214 03:02:50.114079 (+ 774us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:50.114082 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:50.114130 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:50.114200 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:50.115770 (+ 1570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.116269 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:50.116854 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.116978 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:50.119112 (+ 2134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:50.119134 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:50.119136 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:50.119164 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:50.120598 (+ 1434us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:50.121151 (+ 553us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:50.121156 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:50.121158 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:50.121249 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:50.121543 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:50.121546 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:50.121548 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:50.121710 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:50.121789 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:50.122306 (+ 517us) server_negotiation.cc:300] Negotiation successful
1214 03:02:50.122416 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251214 03:02:50.123229 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:50.112745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:46519 (local address 127.1.157.194:40317)
1214 03:02:50.113164 (+ 419us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:50.113221 (+ 57us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:50.113913 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:50.114362 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:50.114370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:50.114744 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:50.115639 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:50.115653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.116361 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:50.116363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:50.116776 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:50.116782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.116945 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:50.117550 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:50.117574 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:50.118945 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:50.120739 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:50.120744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:50.120754 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:50.121048 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:50.121338 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:50.121341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:50.121342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:50.121448 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:50.121816 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:50.121823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:50.122063 (+ 240us) client_negotiation.cc:769] Sending connection context
1214 03:02:50.122266 (+ 203us) client_negotiation.cc:240] Negotiation successful
1214 03:02:50.122545 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20251214 03:02:50.124428 8212 heartbeater.cc:344] Connected to a master server at 127.1.157.254:46519
I20251214 03:02:50.124696 8212 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:50.125293 8212 heartbeater.cc:507] Master 127.1.157.254:46519 requested a full tablet report, sending...
I20251214 03:02:50.126370 7883 ts_manager.cc:194] Registered new tserver with Master: 5b0d20de34f1494eaeddf2384d47b909 (127.1.157.194:37463)
I20251214 03:02:50.126950 7883 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:40317
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:02:50.165563 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:46519
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:50.269147 8219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:50.269424 8219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:50.269505 8219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:50.272854 8219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:50.272935 8219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:50.273041 8219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:50.277411 8219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:50.278487 8219 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:50.279320 8219 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:50.285805 8225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:50.285807 8224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:50.285948 8219 server_base.cc:1047] running on GCE node
W20251214 03:02:50.285807 8227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:50.286425 8219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:50.287002 8219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:50.288164 8219 hybrid_clock.cc:648] HybridClock initialized: now 1765681370288151 us; error 29 us; skew 500 ppm
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:50.290800 8219 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:02:50.291862 8219 webserver.cc:492] Webserver started at http://127.1.157.195:37303/ using document root <none> and password file <none>
I20251214 03:02:50.292409 8219 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:50.292459 8219 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:50.292636 8219 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:50.294358 8219 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "dab2ecd7a9664c3fbf343d8c739abdaf"
format_stamp: "Formatted at 2025-12-14 03:02:50 on dist-test-slave-k7z7"
server_key: "9465d8962f1ad5a8f176ea91f7f3e12c"
server_key_iv: "c552f985105b32c677d75dc8fa1fba86"
server_key_version: "encryptionkey@0"
I20251214 03:02:50.294821 8219 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "dab2ecd7a9664c3fbf343d8c739abdaf"
format_stamp: "Formatted at 2025-12-14 03:02:50 on dist-test-slave-k7z7"
server_key: "9465d8962f1ad5a8f176ea91f7f3e12c"
server_key_iv: "c552f985105b32c677d75dc8fa1fba86"
server_key_version: "encryptionkey@0"
I20251214 03:02:50.298149 8219 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:02:50.300187 8234 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:50.301154 8219 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:50.301304 8219 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "dab2ecd7a9664c3fbf343d8c739abdaf"
format_stamp: "Formatted at 2025-12-14 03:02:50 on dist-test-slave-k7z7"
server_key: "9465d8962f1ad5a8f176ea91f7f3e12c"
server_key_iv: "c552f985105b32c677d75dc8fa1fba86"
server_key_version: "encryptionkey@0"
I20251214 03:02:50.301393 8219 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:50.311594 8219 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:50.314646 8219 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:50.314818 8219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:50.315375 8219 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:50.316284 8219 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:50.316334 8219 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:50.316372 8219 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:50.316388 8219 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:50.325062 8219 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:40155
I20251214 03:02:50.325088 8347 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:40155 every 8 connection(s)
I20251214 03:02:50.325994 8219 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:02:50.330670 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 8219
I20251214 03:02:50.330791 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:02:50.330998 1655 external_mini_cluster.cc:1468] Setting key be4ff2bc0530ff82db5cc0bbddd9cb06
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:02:50.338403 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:50.327727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:41475 (local address 127.1.157.254:46519)
1214 03:02:50.327864 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:50.327868 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:50.328560 (+ 692us) server_negotiation.cc:408] Connection header received
1214 03:02:50.329530 (+ 970us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:50.329534 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:50.329597 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:50.329677 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:50.331439 (+ 1762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.332116 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:50.332982 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.333141 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:50.335295 (+ 2154us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:50.335310 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:50.335313 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:50.335339 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:50.336668 (+ 1329us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:50.337146 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:50.337149 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:50.337150 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:50.337270 (+ 120us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:50.337506 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:50.337508 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:50.337509 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:50.337655 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:50.337727 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:50.338156 (+ 429us) server_negotiation.cc:300] Negotiation successful
1214 03:02:50.338283 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251214 03:02:50.338955 8350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:50.327967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:46519 (local address 127.1.157.195:41475)
1214 03:02:50.328425 (+ 458us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:50.328461 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:50.329192 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:50.329846 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:50.329854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:50.330263 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:50.331251 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:50.331268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.332251 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:50.332255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:50.332828 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:50.332838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.333037 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:50.333662 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:50.333690 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:50.335154 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:50.336773 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:50.336780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:50.336790 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:50.337022 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:50.337346 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:50.337348 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:50.337350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:50.337429 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:50.337726 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:50.337731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:50.337945 (+ 214us) client_negotiation.cc:769] Sending connection context
1214 03:02:50.338107 (+ 162us) client_negotiation.cc:240] Negotiation successful
1214 03:02:50.338307 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":132,"threads_started":1}
I20251214 03:02:50.339887 8348 heartbeater.cc:344] Connected to a master server at 127.1.157.254:46519
I20251214 03:02:50.340111 8348 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:50.340606 8348 heartbeater.cc:507] Master 127.1.157.254:46519 requested a full tablet report, sending...
I20251214 03:02:50.341643 7883 ts_manager.cc:194] Registered new tserver with Master: dab2ecd7a9664c3fbf343d8c739abdaf (127.1.157.195:40155)
I20251214 03:02:50.342175 7883 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:41475
I20251214 03:02:50.344671 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/test-admin.keytab.
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:50.358564 1655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251214 03:02:50.360989 8355 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:50.359800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.1.157.193:34163)
1214 03:02:50.360085 (+ 285us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:50.360091 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:50.360112 (+ 21us) server_negotiation.cc:408] Connection header received
1214 03:02:50.360162 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:50.360167 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:50.360329 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:50.360445 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:50.360862 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54020: BlockingRecv error: recv got EOF from 127.0.0.1:54020 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":114,"threads_started":1}
W20251214 03:02:50.361304 1655 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5cc' of type FILE
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:50.362201 1655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:50.362975 1655 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.193@KRBTEST.COM
I20251214 03:02:50.371667 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:50.363779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.1.157.193:34163)
1214 03:02:50.363893 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:50.363897 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:50.363944 (+ 47us) server_negotiation.cc:408] Connection header received
1214 03:02:50.364037 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:50.364039 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:50.364079 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:50.364142 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:50.364703 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.365427 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:50.366062 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.366290 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:50.368101 (+ 1811us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:50.368126 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:50.368138 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:50.368165 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:50.369948 (+ 1783us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:50.370336 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:50.370341 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:50.370343 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:50.370407 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:50.370669 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:50.370672 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:50.370674 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:50.370893 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:50.371081 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:50.371333 (+ 252us) server_negotiation.cc:300] Negotiation successful
1214 03:02:50.371528 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20251214 03:02:50.375174 8355 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:50.374352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.1.157.193:34163)
1214 03:02:50.374486 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:50.374489 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:50.374538 (+ 49us) server_negotiation.cc:408] Connection header received
1214 03:02:50.374643 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:50.374645 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:50.374687 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:50.374744 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:50.375076 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54036: BlockingRecv error: recv got EOF from 127.0.0.1:54036 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20251214 03:02:50.375268 1655 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5cc' of type FILE
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:50.376266 1655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:50.377072 1655 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.193@KRBTEST.COM
I20251214 03:02:50.384618 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:50.377968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.1.157.193:34163)
1214 03:02:50.378079 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:50.378084 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:50.378136 (+ 52us) server_negotiation.cc:408] Connection header received
1214 03:02:50.378261 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:50.378265 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:50.378305 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:50.378368 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:50.378953 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.379441 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:50.380012 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.380213 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:50.382095 (+ 1882us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:50.382112 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:50.382115 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:50.382138 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:50.383462 (+ 1324us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:50.383811 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:50.383814 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:50.383816 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:50.383851 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:50.384017 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:50.384019 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:50.384020 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:50.384141 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:50.384247 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:50.384408 (+ 161us) server_negotiation.cc:300] Negotiation successful
1214 03:02:50.384500 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20251214 03:02:50.387270 8355 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:02:50.386382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.1.157.193:34163)
1214 03:02:50.386510 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:50.386514 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:50.386577 (+ 63us) server_negotiation.cc:408] Connection header received
1214 03:02:50.386708 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:50.386711 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:50.386750 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:50.386810 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:50.387174 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54066: BlockingRecv error: recv got EOF from 127.0.0.1:54066 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20251214 03:02:50.387414 1655 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765681312865542-1655-0/krb5kdc/krb5cc' of type FILE
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:50.388428 1655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:50.389315 1655 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[7835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.193@KRBTEST.COM
I20251214 03:02:50.396931 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:50.390118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.1.157.193:34163)
1214 03:02:50.390239 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:50.390243 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:50.390297 (+ 54us) server_negotiation.cc:408] Connection header received
1214 03:02:50.390394 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:50.390397 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:50.390434 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:50.390501 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:50.391044 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.391512 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:50.392090 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:50.392218 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:50.394096 (+ 1878us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:50.394114 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:50.394116 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:50.394140 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:50.395537 (+ 1397us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:50.395872 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:50.395875 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:50.395876 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:50.395918 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:50.396069 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:50.396071 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:50.396072 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:50.396193 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:50.396302 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:50.396504 (+ 202us) server_negotiation.cc:300] Negotiation successful
1214 03:02:50.396804 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251214 03:02:50.398036 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 7947
I20251214 03:02:50.403023 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 8083
I20251214 03:02:50.409289 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 8219
I20251214 03:02:50.413787 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 7851
2025-12-14T03:02:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2964 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[8368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[8368](info): set up 2 sockets
Dec 14 03:02:50 dist-test-slave-k7z7 krb5kdc[8368](info): commencing operation
krb5kdc: starting...
W20251214 03:02:52.237545 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.803s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:02:52 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:02:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:02:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:02:52.365079 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:42577
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:42577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:52.468385 8384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:52.468636 8384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:52.468690 8384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:52.472020 8384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:52.472092 8384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:52.472121 8384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:52.472139 8384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:52.472155 8384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:52.476415 8384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:42577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:42577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:52.477478 8384 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:52.478267 8384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:52.483561 8389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:52.483660 8384 server_base.cc:1047] running on GCE node
W20251214 03:02:52.483582 8392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:52.483561 8390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:52.484261 8384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:52.485110 8384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:52.486263 8384 hybrid_clock.cc:648] HybridClock initialized: now 1765681372486251 us; error 26 us; skew 500 ppm
Dec 14 03:02:52 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681372, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:52.488648 8384 init.cc:377] Logged in from keytab as oryx/127.1.157.254@KRBTEST.COM (short username oryx)
I20251214 03:02:52.489771 8384 webserver.cc:492] Webserver started at http://127.1.157.254:39691/ using document root <none> and password file <none>
I20251214 03:02:52.490298 8384 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:52.490355 8384 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:52.490571 8384 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:52.492251 8384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "fc546e7566b6428ca27448a8b619ad2b"
format_stamp: "Formatted at 2025-12-14 03:02:52 on dist-test-slave-k7z7"
server_key: "3aa84955426fe0f504a04ee20bb01ed0"
server_key_iv: "209985f846c022b7b64dfd4ddc1aa545"
server_key_version: "encryptionkey@0"
I20251214 03:02:52.492727 8384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "fc546e7566b6428ca27448a8b619ad2b"
format_stamp: "Formatted at 2025-12-14 03:02:52 on dist-test-slave-k7z7"
server_key: "3aa84955426fe0f504a04ee20bb01ed0"
server_key_iv: "209985f846c022b7b64dfd4ddc1aa545"
server_key_version: "encryptionkey@0"
I20251214 03:02:52.496158 8384 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251214 03:02:52.498317 8399 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:52.499306 8384 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:52.499414 8384 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "fc546e7566b6428ca27448a8b619ad2b"
format_stamp: "Formatted at 2025-12-14 03:02:52 on dist-test-slave-k7z7"
server_key: "3aa84955426fe0f504a04ee20bb01ed0"
server_key_iv: "209985f846c022b7b64dfd4ddc1aa545"
server_key_version: "encryptionkey@0"
I20251214 03:02:52.499508 8384 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:52.508801 8384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:52.514618 8384 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:52.514786 8384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:52.521888 8384 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:42577
I20251214 03:02:52.521888 8451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:42577 every 8 connection(s)
I20251214 03:02:52.522869 8384 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:02:52.525611 8452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:52.530285 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 8384
I20251214 03:02:52.530398 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:02:52.530611 1655 external_mini_cluster.cc:1468] Setting key 1082637f6845cadf2e8a64c8219a34fa
I20251214 03:02:52.531077 8452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: Bootstrap starting.
I20251214 03:02:52.533396 8452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:52.534096 8452 log.cc:826] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:52.535778 8452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: No bootstrap required, opened a new log
Dec 14 03:02:52 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:02:52.538630 8452 raft_consensus.cc:359] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } }
I20251214 03:02:52.538856 8452 raft_consensus.cc:385] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:52.538923 8452 raft_consensus.cc:740] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc546e7566b6428ca27448a8b619ad2b, State: Initialized, Role: FOLLOWER
I20251214 03:02:52.539404 8452 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } }
I20251214 03:02:52.539549 8452 raft_consensus.cc:399] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:02:52.539608 8452 raft_consensus.cc:493] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:02:52.539695 8452 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:52.540597 8452 raft_consensus.cc:515] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } }
I20251214 03:02:52.540958 8452 leader_election.cc:304] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc546e7566b6428ca27448a8b619ad2b; no voters:
I20251214 03:02:52.541265 8452 leader_election.cc:290] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:02:52.541376 8457 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:52.541630 8457 raft_consensus.cc:697] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 1 LEADER]: Becoming Leader. State: Replica: fc546e7566b6428ca27448a8b619ad2b, State: Running, Role: LEADER
I20251214 03:02:52.541970 8457 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } }
I20251214 03:02:52.542454 8452 sys_catalog.cc:565] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:52.543017 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:52.531823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.1.157.254:42577)
1214 03:02:52.532251 (+ 428us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:52.532260 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:52.532293 (+ 33us) server_negotiation.cc:408] Connection header received
1214 03:02:52.532850 (+ 557us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:52.532867 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:52.533155 (+ 288us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:52.533500 (+ 345us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:52.534341 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:52.535101 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:52.535726 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:52.535985 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:52.538206 (+ 2221us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:52.538251 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:52.538261 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:52.538293 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:52.540088 (+ 1795us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:52.540540 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:52.540545 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:52.540549 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:52.540616 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:52.540882 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:52.540886 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:52.540887 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:52.541372 (+ 485us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:52.541606 (+ 234us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:52.541958 (+ 352us) server_negotiation.cc:300] Negotiation successful
1214 03:02:52.542240 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":152,"threads_started":1}
I20251214 03:02:52.544077 8457 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc546e7566b6428ca27448a8b619ad2b. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } }
I20251214 03:02:52.544265 8457 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:52.545526 8458 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } }
I20251214 03:02:52.545610 8458 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: This master's current role is: LEADER
W20251214 03:02:52.547783 8472 catalog_manager.cc:1568] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251214 03:02:52.547855 8472 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251214 03:02:52.547945 8471 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:02:52.548709 8471 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:02:52.554553 8471 catalog_manager.cc:1357] Generated new cluster ID: 4fcd4b0673a74ba5a0393d9fa9c2a01e
I20251214 03:02:52.554648 8471 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:02:52.585835 8471 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:02:52.586844 8471 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:02:52.595944 8471 catalog_manager.cc:6027] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: Generated new TSK 0
I20251214 03:02:52.596742 8471 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:02:52.646114 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42577
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:52.771001 8480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:52.771224 8480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:52.771282 8480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:52.774549 8480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:52.774605 8480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:52.774698 8480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:52.778697 8480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:42577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:52.779645 8480 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:52.780417 8480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:52.787225 8488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:52.787300 8480 server_base.cc:1047] running on GCE node
W20251214 03:02:52.787192 8486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:52.787207 8485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:52.787774 8480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:52.788314 8480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:52.789455 8480 hybrid_clock.cc:648] HybridClock initialized: now 1765681372789445 us; error 32 us; skew 500 ppm
Dec 14 03:02:52 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681372, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:52.791918 8480 init.cc:377] Logged in from keytab as oryx/127.1.157.193@KRBTEST.COM (short username oryx)
I20251214 03:02:52.792884 8480 webserver.cc:492] Webserver started at http://127.1.157.193:42615/ using document root <none> and password file <none>
I20251214 03:02:52.793423 8480 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:52.793470 8480 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:52.793643 8480 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:52.795207 8480 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "74d4903d11c04204a94556ce1adb4088"
format_stamp: "Formatted at 2025-12-14 03:02:52 on dist-test-slave-k7z7"
server_key: "fbaee491e9bd6a1e758b02cd552cca4c"
server_key_iv: "4d582c79a8bfb28847492a0fead724b1"
server_key_version: "encryptionkey@0"
I20251214 03:02:52.795640 8480 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "74d4903d11c04204a94556ce1adb4088"
format_stamp: "Formatted at 2025-12-14 03:02:52 on dist-test-slave-k7z7"
server_key: "fbaee491e9bd6a1e758b02cd552cca4c"
server_key_iv: "4d582c79a8bfb28847492a0fead724b1"
server_key_version: "encryptionkey@0"
I20251214 03:02:52.798791 8480 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:02:52.800930 8495 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:52.801976 8480 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:52.802075 8480 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "74d4903d11c04204a94556ce1adb4088"
format_stamp: "Formatted at 2025-12-14 03:02:52 on dist-test-slave-k7z7"
server_key: "fbaee491e9bd6a1e758b02cd552cca4c"
server_key_iv: "4d582c79a8bfb28847492a0fead724b1"
server_key_version: "encryptionkey@0"
I20251214 03:02:52.802165 8480 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:52.819176 8480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:52.822022 8480 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:52.822192 8480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:52.822707 8480 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:52.823562 8480 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:52.823611 8480 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:52.823649 8480 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:52.823673 8480 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:52.832497 8480 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:36175
I20251214 03:02:52.832525 8608 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:36175 every 8 connection(s)
I20251214 03:02:52.833424 8480 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
Dec 14 03:02:52 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681372, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:02:52.842092 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 8480
I20251214 03:02:52.842201 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:02:52.842406 1655 external_mini_cluster.cc:1468] Setting key d184cebbc39740345fa128e77f06e066
I20251214 03:02:52.845321 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:52.835007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:49543 (local address 127.1.157.254:42577)
1214 03:02:52.835127 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:52.835131 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:52.835730 (+ 599us) server_negotiation.cc:408] Connection header received
1214 03:02:52.836521 (+ 791us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:52.836525 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:52.836585 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:52.836658 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:52.838309 (+ 1651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:52.838811 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:52.839405 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:52.839589 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:52.841702 (+ 2113us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:52.841725 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:52.841728 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:52.841755 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:52.843140 (+ 1385us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:52.843743 (+ 603us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:52.843747 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:52.843749 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:52.843803 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:52.844237 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:52.844240 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:52.844242 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:52.844427 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:52.844523 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:52.844989 (+ 466us) server_negotiation.cc:300] Negotiation successful
1214 03:02:52.845121 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251214 03:02:52.846025 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:52.835223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.193:49543)
1214 03:02:52.835603 (+ 380us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:52.835636 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:52.836332 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:52.836828 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:52.836836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:52.837242 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:52.838161 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:52.838175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:52.838921 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:52.838924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:52.839300 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:52.839307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:52.839514 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:52.840061 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:52.840082 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:52.841563 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:52.843254 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:52.843259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:52.843272 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:52.843581 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:52.843922 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:52.843925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:52.843926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:52.844116 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:52.844524 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:52.844530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:52.844768 (+ 238us) client_negotiation.cc:769] Sending connection context
1214 03:02:52.844940 (+ 172us) client_negotiation.cc:240] Negotiation successful
1214 03:02:52.845161 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20251214 03:02:52.847096 8609 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42577
I20251214 03:02:52.847354 8609 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:52.847842 8609 heartbeater.cc:507] Master 127.1.157.254:42577 requested a full tablet report, sending...
I20251214 03:02:52.849257 8416 ts_manager.cc:194] Registered new tserver with Master: 74d4903d11c04204a94556ce1adb4088 (127.1.157.193:36175)
I20251214 03:02:52.850384 8416 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:49543
WARNING: no policy specified for oryx/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:02:52.884289 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42577
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:52.987589 8616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:52.987808 8616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:52.987866 8616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:52.991139 8616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:52.991199 8616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:52.991302 8616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:52.995523 8616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:42577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:52.996547 8616 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:52.997432 8616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251214 03:02:53.004494 8616 server_base.cc:1047] running on GCE node
W20251214 03:02:53.004374 8622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:53.004520 8621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:53.004374 8624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:53.004981 8616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:53.005544 8616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:53.006685 8616 hybrid_clock.cc:648] HybridClock initialized: now 1765681373006672 us; error 26 us; skew 500 ppm
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:53.009410 8616 init.cc:377] Logged in from keytab as oryx/127.1.157.194@KRBTEST.COM (short username oryx)
I20251214 03:02:53.010475 8616 webserver.cc:492] Webserver started at http://127.1.157.194:44773/ using document root <none> and password file <none>
I20251214 03:02:53.011006 8616 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:53.011067 8616 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:53.011252 8616 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:53.012907 8616 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "af3d55d43111412b82f233739185d287"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "2513802fbbc6010cb28de5cd6d0d8f0c"
server_key_iv: "ca161b52d70a63cb21c1a34822ce60b3"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.013370 8616 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "af3d55d43111412b82f233739185d287"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "2513802fbbc6010cb28de5cd6d0d8f0c"
server_key_iv: "ca161b52d70a63cb21c1a34822ce60b3"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.016620 8616 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251214 03:02:53.018733 8631 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.019675 8616 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251214 03:02:53.019769 8616 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "af3d55d43111412b82f233739185d287"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "2513802fbbc6010cb28de5cd6d0d8f0c"
server_key_iv: "ca161b52d70a63cb21c1a34822ce60b3"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.019850 8616 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:53.039899 8616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:53.042986 8616 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:53.043171 8616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:53.043792 8616 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:53.044749 8616 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:53.044799 8616 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.044852 8616 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:53.044879 8616 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.053722 8616 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:43857
I20251214 03:02:53.053747 8744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:43857 every 8 connection(s)
I20251214 03:02:53.054656 8616 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:02:53.059679 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 8616
I20251214 03:02:53.059792 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:02:53.060053 1655 external_mini_cluster.cc:1468] Setting key 0f39aa0591ec2b2698a7cfe74727a526
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:02:53.067808 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.056428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:56175 (local address 127.1.157.254:42577)
1214 03:02:53.056591 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.056596 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.057268 (+ 672us) server_negotiation.cc:408] Connection header received
1214 03:02:53.058114 (+ 846us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.058118 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.058185 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.058255 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.059966 (+ 1711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.060626 (+ 660us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.061387 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.061561 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.063869 (+ 2308us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.063887 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.063889 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.063917 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.065774 (+ 1857us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.066312 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.066315 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.066317 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.066370 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.066677 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.066682 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.066684 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.066859 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.066957 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.067477 (+ 520us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.067625 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for oryx/127.1.157.195@KRBTEST.COM; defaulting to no policy
I20251214 03:02:53.068490 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.056664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.194:56175)
1214 03:02:53.057104 (+ 440us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.057141 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.057911 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.058430 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.058438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.058820 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.059779 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.059794 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.060776 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.060779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.061267 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.061278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.061546 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.062229 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.062255 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.063714 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.065901 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.065908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.065922 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.066193 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.066462 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.066466 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.066467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.066582 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.066969 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.066977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.067224 (+ 247us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.067433 (+ 209us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.067722 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":94,"threads_started":1}
I20251214 03:02:53.069700 8745 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42577
I20251214 03:02:53.069934 8745 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:53.070504 8745 heartbeater.cc:507] Master 127.1.157.254:42577 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.195@KRBTEST.COM" created.
I20251214 03:02:53.071666 8416 ts_manager.cc:194] Registered new tserver with Master: af3d55d43111412b82f233739185d287 (127.1.157.194:43857)
I20251214 03:02:53.072240 8416 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:56175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:02:53.109881 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42577
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:53.212081 8752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:53.212320 8752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:53.212399 8752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:53.215726 8752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:53.215802 8752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:53.215906 8752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:53.220214 8752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:42577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:53.221282 8752 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:53.222084 8752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:53.228277 8760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:53.228271 8758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:53.228263 8757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:53.228540 8752 server_base.cc:1047] running on GCE node
I20251214 03:02:53.228920 8752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:53.229458 8752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:53.230605 8752 hybrid_clock.cc:648] HybridClock initialized: now 1765681373230595 us; error 23 us; skew 500 ppm
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:53.233071 8752 init.cc:377] Logged in from keytab as oryx/127.1.157.195@KRBTEST.COM (short username oryx)
I20251214 03:02:53.234119 8752 webserver.cc:492] Webserver started at http://127.1.157.195:46097/ using document root <none> and password file <none>
I20251214 03:02:53.234613 8752 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:53.234658 8752 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:53.234803 8752 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:53.236346 8752 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "7dd607b9ac1a4c779da1359d96d5b5f9"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "247135edee0d7e4a75e80edeb661a519"
server_key_iv: "861bf1e8cf2a35248d9e3e7918746d5d"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.236752 8752 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "7dd607b9ac1a4c779da1359d96d5b5f9"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "247135edee0d7e4a75e80edeb661a519"
server_key_iv: "861bf1e8cf2a35248d9e3e7918746d5d"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.239848 8752 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:02:53.241948 8767 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.242961 8752 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20251214 03:02:53.243059 8752 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "7dd607b9ac1a4c779da1359d96d5b5f9"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "247135edee0d7e4a75e80edeb661a519"
server_key_iv: "861bf1e8cf2a35248d9e3e7918746d5d"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.243153 8752 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:53.254031 8752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:53.256817 8752 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:53.256973 8752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:53.257512 8752 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:53.258354 8752 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:53.258401 8752 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.258447 8752 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:53.258467 8752 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.267696 8752 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:41383
I20251214 03:02:53.267719 8880 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:41383 every 8 connection(s)
I20251214 03:02:53.268684 8752 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:02:53.275449 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 8752
I20251214 03:02:53.275619 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:02:53.276053 1655 external_mini_cluster.cc:1468] Setting key 0e5b1fc7c42754605fc224f49c4b8f33
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:02:53.282027 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.270860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:58323 (local address 127.1.157.254:42577)
1214 03:02:53.271103 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.271107 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.271918 (+ 811us) server_negotiation.cc:408] Connection header received
1214 03:02:53.272826 (+ 908us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.272830 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.272892 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.272968 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.274778 (+ 1810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.275298 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.275971 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.276154 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.278995 (+ 2841us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.279011 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.279014 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.279041 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.280306 (+ 1265us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.280774 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.280777 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.280778 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.280816 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.281032 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.281034 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.281036 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.281230 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.281318 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.281792 (+ 474us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.281901 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251214 03:02:53.282676 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.271128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.195:58323)
1214 03:02:53.271762 (+ 634us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.271800 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.272604 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.273193 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.273208 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.273641 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.274606 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.274623 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.275422 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.275426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.275855 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.275864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.276106 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.277041 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.277080 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.278837 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.280429 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.280434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.280458 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.280698 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.280876 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.280878 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.280880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.280972 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.281321 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.281326 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.281582 (+ 256us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.281757 (+ 175us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.281977 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":132,"threads_started":1}
I20251214 03:02:53.283706 8881 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42577
I20251214 03:02:53.283929 8881 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:53.284425 8881 heartbeater.cc:507] Master 127.1.157.254:42577 requested a full tablet report, sending...
I20251214 03:02:53.285480 8416 ts_manager.cc:194] Registered new tserver with Master: 7dd607b9ac1a4c779da1359d96d5b5f9 (127.1.157.195:41383)
I20251214 03:02:53.286039 8416 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:58323
I20251214 03:02:53.290447 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.157.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.1.157.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.1.157.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.1.157.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:02:53.337929 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.253:44661
--webserver_interface=127.1.157.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:42577,127.1.157.253:44661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:53.450917 8888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:53.451145 8888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:53.451205 8888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:53.454381 8888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:53.454435 8888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:53.454458 8888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:53.454474 8888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:53.454501 8888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:53.458671 8888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:42577,127.1.157.253:44661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.253:44661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.157.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
W20251214 03:02:53.458856 8888 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.157.254:42577,127.1.157.253:44661'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251214 03:02:53.459661 8888 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:53.460427 8888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:53.466032 8893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:53.466090 8896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:53.466096 8894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:53.466399 8888 server_base.cc:1047] running on GCE node
I20251214 03:02:53.466904 8888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:53.467772 8888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:53.468972 8888 hybrid_clock.cc:648] HybridClock initialized: now 1765681373468960 us; error 28 us; skew 500 ppm
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:53.471644 8888 init.cc:377] Logged in from keytab as oryx/127.1.157.253@KRBTEST.COM (short username oryx)
I20251214 03:02:53.472681 8888 webserver.cc:492] Webserver started at http://127.1.157.253:44731/ using document root <none> and password file <none>
I20251214 03:02:53.473256 8888 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:53.473310 8888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:53.473485 8888 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:53.475092 8888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data/instance:
uuid: "d5053326dfe84c4a97d163a30e8c4d60"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "dcb0dde710d5fde02881231ee4673f76"
server_key_iv: "16fb8284d458726df3ef2fe1b90c817f"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.475540 8888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal/instance:
uuid: "d5053326dfe84c4a97d163a30e8c4d60"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "dcb0dde710d5fde02881231ee4673f76"
server_key_iv: "16fb8284d458726df3ef2fe1b90c817f"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.478736 8888 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:02:53.480727 8903 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.481719 8888 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:53.481830 8888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal
uuid: "d5053326dfe84c4a97d163a30e8c4d60"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "dcb0dde710d5fde02881231ee4673f76"
server_key_iv: "16fb8284d458726df3ef2fe1b90c817f"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.481909 8888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:53.501288 8888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:53.504321 8888 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:53.504494 8888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:53.511627 8888 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.253:44661
I20251214 03:02:53.511624 8955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.253:44661 every 8 connection(s)
I20251214 03:02:53.512629 8888 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data/info.pb
I20251214 03:02:53.513774 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 8888
I20251214 03:02:53.515832 8956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:53.519203 8956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } has no permanent_uuid. Determining permanent_uuid...
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.157.253@KRBTEST.COM
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.253@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:02:53.527926 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.515101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.1.157.253:44661)
1214 03:02:53.515521 (+ 420us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.515530 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.515565 (+ 35us) server_negotiation.cc:408] Connection header received
1214 03:02:53.516635 (+ 1070us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.516658 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.516950 (+ 292us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.517291 (+ 341us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.518178 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.518928 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.519735 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.520024 (+ 289us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.522559 (+ 2535us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.522615 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.522625 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.522659 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.524601 (+ 1942us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.525066 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.525072 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.525077 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.525206 (+ 129us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.525434 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.525437 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.525439 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.525766 (+ 327us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.525954 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.526830 (+ 876us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.527166 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
I20251214 03:02:53.530351 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.520832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56100)
1214 03:02:53.521238 (+ 406us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.521277 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.521402 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.522175 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.522182 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.522607 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.522923 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.522931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.523883 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.523887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.524499 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.524508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.524616 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.525230 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.525245 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.526832 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.528811 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.528818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.528823 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.529066 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.529356 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.529359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.529361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.529415 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.529859 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.529865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.529969 (+ 104us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.530078 (+ 109us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.530210 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":118,"threads_started":1}
I20251214 03:02:53.530436 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.520951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.1.157.254:42577)
1214 03:02:53.521105 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.521110 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.521932 (+ 822us) server_negotiation.cc:408] Connection header received
1214 03:02:53.521979 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.521982 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.522028 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.522101 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.523035 (+ 934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.523775 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.524615 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.524799 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.526962 (+ 2163us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.526981 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.526984 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.527013 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.528692 (+ 1679us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.529162 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.529196 (+ 34us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.529198 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.529256 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.529497 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.529500 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.529503 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.529702 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.529797 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.530154 (+ 357us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.530288 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251214 03:02:53.531373 8956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } has no permanent_uuid. Determining permanent_uuid...
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.253@KRBTEST.COM for oryx/127.1.157.253@KRBTEST.COM
I20251214 03:02:53.538195 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.531825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.0.0.1:41918)
1214 03:02:53.531934 (+ 109us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.531947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.532015 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.532204 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.532207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.532369 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.532606 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.532613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.533235 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.533239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.533763 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.533771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.533874 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.534339 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.534353 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.535551 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.537080 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.537082 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.537084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.537299 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.537516 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.537519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.537520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.537565 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.537816 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.537818 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.537887 (+ 69us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.537957 (+ 70us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.538056 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20251214 03:02:53.538226 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.531874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.1.157.253:44661)
1214 03:02:53.531969 (+ 95us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.531972 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.531984 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:02:53.532092 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.532094 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.532128 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.532186 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.532697 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.533136 (+ 439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.533853 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.534031 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.535657 (+ 1626us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.535672 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.535674 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.535695 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.536981 (+ 1286us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.537374 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.537377 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.537378 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.537412 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.537635 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.537637 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.537638 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.537760 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.537821 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.537985 (+ 164us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.538109 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":38}
I20251214 03:02:53.540643 8956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Bootstrap starting.
I20251214 03:02:53.543207 8956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:53.543963 8956 log.cc:826] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:53.545820 8956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: No bootstrap required, opened a new log
I20251214 03:02:53.548324 8956 raft_consensus.cc:359] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } }
I20251214 03:02:53.548501 8956 raft_consensus.cc:385] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:53.548566 8956 raft_consensus.cc:740] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5053326dfe84c4a97d163a30e8c4d60, State: Initialized, Role: FOLLOWER
I20251214 03:02:53.549122 8956 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } }
I20251214 03:02:53.549647 8962 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } } }
I20251214 03:02:53.549758 8962 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 03:02:53.550272 8956 sys_catalog.cc:565] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: configured and running, proceeding with master startup.
W20251214 03:02:53.554699 8973 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251214 03:02:53.554781 8973 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251214 03:02:53.556803 8888 master_runner.cc:428] Detected that this master d5053326dfe84c4a97d163a30e8c4d60 is joining an existing cluster
I20251214 03:02:53.556866 8888 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.157.253:44661
I20251214 03:02:53.569630 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.558893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56112)
1214 03:02:53.559310 (+ 417us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.559324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.559450 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.559854 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.559857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.560481 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.561325 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.561345 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.562625 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.562630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.563144 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.563155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.563319 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.564108 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.564149 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.564549 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.566961 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.566964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.566966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.567261 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.568305 (+ 1044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.568308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.568309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.568367 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.569017 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.569021 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.569163 (+ 142us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.569314 (+ 151us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.569458 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"spinlock_wait_cycles":768,"thread_start_us":127,"threads_started":1}
I20251214 03:02:53.569844 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.558978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.1.157.254:42577)
1214 03:02:53.559116 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.559121 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.559406 (+ 285us) server_negotiation.cc:408] Connection header received
1214 03:02:53.559679 (+ 273us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.559683 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.559740 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.559831 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.561549 (+ 1718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.562389 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.563310 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.563546 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.564705 (+ 1159us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.564733 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.564737 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.564770 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.566680 (+ 1910us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.567487 (+ 807us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.567491 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.567493 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.567559 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.568503 (+ 944us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.568509 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.568511 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.568851 (+ 340us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.569003 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.569512 (+ 509us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.569679 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681372, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.254@KRBTEST.COM for oryx/127.1.157.253@KRBTEST.COM
I20251214 03:02:53.583588 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.559067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.0.0.1:41922)
1214 03:02:53.559310 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.559324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.559450 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.559881 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.559884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.577350 (+ 17466us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.577839 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.577845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.578415 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.578417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.578897 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.578905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.579019 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.579418 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.579431 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.579700 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.582027 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.582030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.582032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.582513 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.582725 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.582727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.582729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.582774 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.583060 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.583063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.583138 (+ 75us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.583275 (+ 137us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.583399 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"spinlock_wait_cycles":19456,"thread_start_us":121,"threads_started":1}
I20251214 03:02:53.584013 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.558970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.1.157.253:44661)
1214 03:02:53.559102 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.559107 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.559389 (+ 282us) server_negotiation.cc:408] Connection header received
1214 03:02:53.559676 (+ 287us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.559680 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.559726 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.559764 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.577930 (+ 18166us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.578317 (+ 387us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.579017 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.579210 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.579807 (+ 597us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.579825 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.579827 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.579847 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.581927 (+ 2080us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.582606 (+ 679us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.582609 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.582611 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.582648 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.582842 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.582845 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.582846 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.582977 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.583080 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.583791 (+ 711us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.583889 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251214 03:02:53.585131 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.575293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.0.0.1:41924)
1214 03:02:53.575681 (+ 388us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.575724 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.575871 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.576180 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.576185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.576605 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.576829 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.576836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.577777 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.577782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.578372 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.578382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.578514 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.579360 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.579377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.581146 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.583285 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.583295 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.583301 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.583599 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.583948 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.583951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.583953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.584007 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.584374 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.584381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.584651 (+ 270us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.584797 (+ 146us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.584962 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20251214 03:02:53.585897 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.575374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.1.157.253:44661)
1214 03:02:53.575687 (+ 313us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.575692 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.575794 (+ 102us) server_negotiation.cc:408] Connection header received
1214 03:02:53.576003 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.576006 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.576053 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.576126 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.576948 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.577641 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.578482 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.578666 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.581308 (+ 2642us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.581325 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.581329 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.581353 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.583161 (+ 1808us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.583809 (+ 648us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.583812 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.583815 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.583860 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.584091 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.584094 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.584096 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.584241 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.584321 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.585552 (+ 1231us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.585663 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20251214 03:02:53.587088 8416 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.1.157.253:44661: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60"
member_type: NON_VOTER
last_known_addr {
host: "127.1.157.253"
port: 44661
}
attrs {
promote: true
}
}
dest_uuid: "fc546e7566b6428ca27448a8b619ad2b"
cas_config_opid_index: -1
I20251214 03:02:53.587772 8416 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.1.157.252@KRBTEST.COM; defaulting to no policy
I20251214 03:02:53.589761 8985 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d5053326dfe84c4a97d163a30e8c4d60 (127.1.157.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } } }
I20251214 03:02:53.590361 8930 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:53.591843 8985 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.1.157.253:44661
I20251214 03:02:53.592052 8985 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } } }
I20251214 03:02:53.592099 8930 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Refusing update from remote peer fc546e7566b6428ca27448a8b619ad2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251214 03:02:53.592135 8985 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:53.592278 8888 master.cc:561] Master@127.1.157.253:44661 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.252@KRBTEST.COM" created.
I20251214 03:02:53.593145 8984 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 03:02:53.594916 8403 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20251214 03:02:53.595824 8403 consensus_peers.cc:597] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b -> Peer d5053326dfe84c4a97d163a30e8c4d60 (127.1.157.253:44661): Couldn't send request to peer d5053326dfe84c4a97d163a30e8c4d60. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251214 03:02:53.596495 8888 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:53.596729 8888 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:53.596807 8888 tablet_replica.cc:333] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: stopping tablet replica
I20251214 03:02:53.600557 8888 master.cc:583] Master@127.1.157.253:44661 shutdown complete.
I20251214 03:02:53.600598 8888 master_runner.cc:305] Clearing existing system tablet
I20251214 03:02:53.600798 8888 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:53.600845 8888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:53.602021 8888 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:53.602766 8990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.602991 8888 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:53.603121 8888 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:53.603161 8888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal
uuid: "d5053326dfe84c4a97d163a30e8c4d60"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "dcb0dde710d5fde02881231ee4673f76"
server_key_iv: "16fb8284d458726df3ef2fe1b90c817f"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.605225 8888 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:53.607354 8888 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251214 03:02:53.607430 8888 log.cc:1199] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251214 03:02:53.607630 8888 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Deleting consensus metadata
I20251214 03:02:53.608299 8888 master_runner.cc:315] Copying system tablet from 127.1.157.254:42577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.1.157.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:02:53.609376 8888 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: tablet copy: Beginning tablet copy session from remote peer at address 127.1.157.254:42577
I20251214 03:02:53.616231 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.609930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.1.157.254:42577)
1214 03:02:53.610067 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.610071 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.610261 (+ 190us) server_negotiation.cc:408] Connection header received
1214 03:02:53.610383 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.610387 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.610434 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.610498 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.611033 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.611792 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.612414 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.612594 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.613546 (+ 952us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.613569 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.613572 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.613598 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.614889 (+ 1291us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.615253 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.615256 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.615257 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.615299 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.615491 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.615495 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.615497 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.615651 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.615735 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.615968 (+ 233us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.616076 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251214 03:02:53.616232 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.609849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56114)
1214 03:02:53.610181 (+ 332us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.610192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.610300 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.610523 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.610526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.610708 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.610909 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.610916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.611901 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.611904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.612312 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.612318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.612421 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.613036 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.613049 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.613426 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.614993 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.614997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.614999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.615163 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.615374 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.615377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.615378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.615420 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.615733 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.615736 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.615819 (+ 83us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.615904 (+ 85us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.616048 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":83,"threads_started":1}
I20251214 03:02:53.617046 8436 tablet_copy_service.cc:140] P fc546e7566b6428ca27448a8b619ad2b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d5053326dfe84c4a97d163a30e8c4d60 ({username='oryx', principal='oryx/127.1.157.253@KRBTEST.COM'} at 127.0.0.1:56114)
I20251214 03:02:53.617228 8436 tablet_copy_service.cc:161] P fc546e7566b6428ca27448a8b619ad2b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d5053326dfe84c4a97d163a30e8c4d60 at {username='oryx', principal='oryx/127.1.157.253@KRBTEST.COM'} at 127.0.0.1:56114: session id = d5053326dfe84c4a97d163a30e8c4d60-00000000000000000000000000000000
WARNING: no policy specified for HTTP/127.1.157.252@KRBTEST.COM; defaulting to no policy
I20251214 03:02:53.618547 8436 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 03:02:53.619987 8888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.252@KRBTEST.COM" created.
I20251214 03:02:53.621959 8888 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: tablet copy: Starting download of 0 data blocks...
I20251214 03:02:53.622121 8888 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: tablet copy: Starting download of 1 WAL segments...
I20251214 03:02:53.623833 8888 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 03:02:53.625451 8436 tablet_copy_service.cc:342] P fc546e7566b6428ca27448a8b619ad2b: Request end of tablet copy session d5053326dfe84c4a97d163a30e8c4d60-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.157.253@KRBTEST.COM'} at 127.0.0.1:56114
I20251214 03:02:53.625567 8436 tablet_copy_service.cc:434] P fc546e7566b6428ca27448a8b619ad2b: ending tablet copy session d5053326dfe84c4a97d163a30e8c4d60-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d5053326dfe84c4a97d163a30e8c4d60
W20251214 03:02:53.629976 8888 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251214 03:02:53.632126 8888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:53.633951 9002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:53.633999 8999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:53.634158 8888 server_base.cc:1047] running on GCE node
W20251214 03:02:53.634366 9000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:53.634683 8888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:53.635023 8888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.1.157.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:02:53.636181 8888 hybrid_clock.cc:648] HybridClock initialized: now 1765681373636181 us; error 42 us; skew 500 ppm
I20251214 03:02:53.636452 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.252:43123
--webserver_interface=127.1.157.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:42577,127.1.157.253:44661,127.1.157.252:43123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:53.637913 8888 init.cc:377] Logged in from keytab as oryx/127.1.157.253@KRBTEST.COM (short username oryx)
I20251214 03:02:53.638407 8888 webserver.cc:492] Webserver started at http://127.1.157.253:38391/ using document root <none> and password file <none>
I20251214 03:02:53.638607 8888 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:53.638646 8888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:53.639940 8888 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:53.640937 9010 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.641242 8888 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:53.641355 8888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal
uuid: "d5053326dfe84c4a97d163a30e8c4d60"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "dcb0dde710d5fde02881231ee4673f76"
server_key_iv: "16fb8284d458726df3ef2fe1b90c817f"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.641505 8888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:53.665422 8888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:53.667475 8888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:53.672375 8888 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.253:44661
I20251214 03:02:53.672408 9062 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.253:44661 every 8 connection(s)
I20251214 03:02:53.673125 8888 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-1/data/info.pb
I20251214 03:02:53.675246 9063 sys_catalog.cc:263] Verifying existing consensus state
I20251214 03:02:53.676175 9063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Bootstrap starting.
I20251214 03:02:53.685938 9063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 03:02:53.686213 9063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Bootstrap complete.
I20251214 03:02:53.686775 9063 raft_consensus.cc:359] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } }
I20251214 03:02:53.686897 9063 raft_consensus.cc:740] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d5053326dfe84c4a97d163a30e8c4d60, State: Initialized, Role: LEARNER
I20251214 03:02:53.686998 9063 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } }
I20251214 03:02:53.687314 9066 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } } }
I20251214 03:02:53.687436 9063 sys_catalog.cc:565] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:53.687474 9066 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: This master's current role is: LEARNER
I20251214 03:02:53.691602 9077 catalog_manager.cc:1269] Loaded cluster ID: 4fcd4b0673a74ba5a0393d9fa9c2a01e
I20251214 03:02:53.691663 9077 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: loading cluster ID for follower catalog manager: success
I20251214 03:02:53.693782 9077 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: acquiring CA information for follower catalog manager: success
I20251214 03:02:53.695116 9077 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251214 03:02:53.750528 9005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:53.750766 9005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:53.750837 9005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:53.754124 9005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:02:53.754191 9005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:53.754230 9005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:53.754261 9005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:02:53.754293 9005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:02:53.758634 9005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:42577,127.1.157.253:44661,127.1.157.252:43123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.252:43123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.157.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:53.759683 9005 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:53.760519 9005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:53.765990 9082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:53.766067 9083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:53.766141 9005 server_base.cc:1047] running on GCE node
W20251214 03:02:53.765998 9085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:53.766783 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:53.767740 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:53.768918 9005 hybrid_clock.cc:648] HybridClock initialized: now 1765681373768898 us; error 36 us; skew 500 ppm
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:53.771765 9005 init.cc:377] Logged in from keytab as oryx/127.1.157.252@KRBTEST.COM (short username oryx)
I20251214 03:02:53.772822 9005 webserver.cc:492] Webserver started at http://127.1.157.252:32819/ using document root <none> and password file <none>
I20251214 03:02:53.773397 9005 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:53.773449 9005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:53.773630 9005 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:02:53.775286 9005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data/instance:
uuid: "e930f3dd0c0a498d913e4239a7b82416"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "c08455fd0ede36a7ececfe1433dcc676"
server_key_iv: "a99802d0d52db509cbbbb65697e399ef"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.775753 9005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal/instance:
uuid: "e930f3dd0c0a498d913e4239a7b82416"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "c08455fd0ede36a7ececfe1433dcc676"
server_key_iv: "a99802d0d52db509cbbbb65697e399ef"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.779048 9005 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:02:53.781154 9092 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.782153 9005 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:53.782254 9005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal
uuid: "e930f3dd0c0a498d913e4239a7b82416"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "c08455fd0ede36a7ececfe1433dcc676"
server_key_iv: "a99802d0d52db509cbbbb65697e399ef"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.782339 9005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:53.812501 9005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:53.815068 9005 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:53.815235 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:53.821691 9005 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.252:43123
I20251214 03:02:53.821695 9144 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.252:43123 every 8 connection(s)
I20251214 03:02:53.822598 9005 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data/info.pb
I20251214 03:02:53.825156 9145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:53.828691 9145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } has no permanent_uuid. Determining permanent_uuid...
I20251214 03:02:53.832712 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 9005
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.252@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.157.252@KRBTEST.COM
I20251214 03:02:53.841638 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.830393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.1.157.254:42577)
1214 03:02:53.830589 (+ 196us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.830594 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.831174 (+ 580us) server_negotiation.cc:408] Connection header received
1214 03:02:53.831878 (+ 704us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.831881 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.831932 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.832007 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.833738 (+ 1731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.834484 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.835325 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.835524 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.837766 (+ 2242us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.837792 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.837796 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.837831 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.839413 (+ 1582us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.840012 (+ 599us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.840016 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.840018 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.840083 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.840392 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.840395 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.840398 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.840609 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.840686 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.841393 (+ 707us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.841507 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20251214 03:02:53.842434 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.830612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56128)
1214 03:02:53.831044 (+ 432us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.831079 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.831688 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.832190 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.832199 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.832746 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.833545 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.833563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.834656 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.834661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.835219 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.835227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.835406 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.836001 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.836026 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.837628 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.839558 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.839564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.839578 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.839906 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.840168 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.840172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.840174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.840309 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.840705 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.840714 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.841031 (+ 317us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.841325 (+ 294us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.841647 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
I20251214 03:02:53.843650 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.833629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.1.157.252:43123)
1214 03:02:53.833946 (+ 317us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.833951 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.833969 (+ 18us) server_negotiation.cc:408] Connection header received
1214 03:02:53.834002 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.834005 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.834132 (+ 127us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.834287 (+ 155us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.835168 (+ 881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.836144 (+ 976us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.836911 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.837224 (+ 313us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.839564 (+ 2340us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.839588 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.839600 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.839622 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.841630 (+ 2008us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.842263 (+ 633us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.842268 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.842270 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.842327 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.842615 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.842618 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.842620 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.842920 (+ 300us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.843083 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.843330 (+ 247us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.843447 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":143,"threads_started":1}
I20251214 03:02:53.843815 9145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } has no permanent_uuid. Determining permanent_uuid...
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.252@KRBTEST.COM for oryx/127.1.157.253@KRBTEST.COM
I20251214 03:02:53.852658 8609 heartbeater.cc:499] Master 127.1.157.254:42577 was elected leader, sending a full tablet report...
I20251214 03:02:53.853199 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.844398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.1.157.253:44661)
1214 03:02:53.844692 (+ 294us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.844696 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.844711 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:02:53.844749 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.844752 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.844865 (+ 113us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.844956 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.845606 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.846292 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.847038 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.847292 (+ 254us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.849134 (+ 1842us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.849159 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.849162 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.849236 (+ 74us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.850936 (+ 1700us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.851701 (+ 765us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.851705 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.851706 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.851766 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.852112 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.852115 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.852117 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.852335 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.852455 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.852861 (+ 406us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.853012 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
I20251214 03:02:53.853284 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.844289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.0.0.1:41932)
1214 03:02:53.844411 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.844425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.844500 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.845006 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.845009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.845228 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.845490 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.845499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.846429 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.846433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.846830 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.846838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.846937 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.847439 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.847452 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.848919 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.851130 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.851135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.851137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.851555 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.851937 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.851940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.851942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.852005 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.852481 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.852485 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.852631 (+ 146us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.852784 (+ 153us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.853002 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251214 03:02:53.854655 9145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } has no permanent_uuid. Determining permanent_uuid...
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.252@KRBTEST.COM for oryx/127.1.157.252@KRBTEST.COM
I20251214 03:02:53.862932 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.855209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.0.0.1:58832)
1214 03:02:53.855463 (+ 254us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.855481 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.855557 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.855893 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.855896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.856073 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.856377 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.856384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.857203 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.857208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.857586 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.857595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.857711 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.858288 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.858303 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.859568 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.861629 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.861633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.861634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.861796 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.862058 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.862061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.862062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.862111 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.862494 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.862497 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.862580 (+ 83us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.862666 (+ 86us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.862777 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20251214 03:02:53.863075 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.855335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.1.157.252:43123)
1214 03:02:53.855506 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.855511 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.855530 (+ 19us) server_negotiation.cc:408] Connection header received
1214 03:02:53.855646 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.855650 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.855714 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.855823 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.856541 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.857031 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.857717 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.857869 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.859712 (+ 1843us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.859733 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.859737 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.859762 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.861500 (+ 1738us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.861900 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.861903 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.861904 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.861946 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.862198 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.862200 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.862202 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.862387 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.862466 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.862779 (+ 313us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.862883 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251214 03:02:53.867231 9145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: Bootstrap starting.
I20251214 03:02:53.869822 9145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:53.870518 9145 log.cc:826] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:53.872385 9145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: No bootstrap required, opened a new log
I20251214 03:02:53.874856 9145 raft_consensus.cc:359] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } }
I20251214 03:02:53.875025 9145 raft_consensus.cc:385] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:53.875057 9145 raft_consensus.cc:740] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e930f3dd0c0a498d913e4239a7b82416, State: Initialized, Role: FOLLOWER
I20251214 03:02:53.875442 9145 consensus_queue.cc:260] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } }
I20251214 03:02:53.875869 9151 sys_catalog.cc:455] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } } }
I20251214 03:02:53.875977 9151 sys_catalog.cc:458] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 03:02:53.876485 9145 sys_catalog.cc:565] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: configured and running, proceeding with master startup.
W20251214 03:02:53.880982 9162 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251214 03:02:53.881078 9162 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251214 03:02:53.884083 9005 master_runner.cc:428] Detected that this master e930f3dd0c0a498d913e4239a7b82416 is joining an existing cluster
I20251214 03:02:53.884143 9005 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.157.252:43123
I20251214 03:02:53.900082 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.897611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56140)
1214 03:02:53.897730 (+ 119us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.897744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.897821 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.898169 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.898172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.898188 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:53.898375 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.898381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.899071 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.899075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.899647 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.899654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.899756 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.899827 (+ 71us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.899879 (+ 52us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.899942 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251214 03:02:53.900442 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.897674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.1.157.254:42577)
1214 03:02:53.897983 (+ 309us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.897986 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.897995 (+ 9us) server_negotiation.cc:408] Connection header received
1214 03:02:53.898035 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.898037 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.898080 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.898150 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:53.898469 (+ 319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.898979 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.899749 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.900151 (+ 402us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.900231 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.900281 (+ 50us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.900354 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":122,"threads_started":1}
I20251214 03:02:53.905078 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.901467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.0.0.1:41950)
1214 03:02:53.902082 (+ 615us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.902095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.902162 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.902594 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.902597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.902617 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:53.902814 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.902821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.903912 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.903915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.904776 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.904787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.904853 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.904867 (+ 14us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.904898 (+ 31us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.904960 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530}
I20251214 03:02:53.906322 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.901724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.1.157.253:44661)
1214 03:02:53.902346 (+ 622us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.902350 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.902362 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:02:53.902397 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.902400 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.902455 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.902905 (+ 450us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:53.902982 (+ 77us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.903791 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.905384 (+ 1593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.905844 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.905996 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.906045 (+ 49us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.906189 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":193,"threads_started":1}
I20251214 03:02:53.907853 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.886344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.0.0.1:41942)
1214 03:02:53.886761 (+ 417us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.886772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.886885 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.887223 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.887225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.901444 (+ 14219us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.901738 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.901748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.902680 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.902684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.903263 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.903273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.903367 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.903935 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.903950 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.904362 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.906474 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.906477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.906480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.906674 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.906909 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.906912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.906914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.906965 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.907346 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.907349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.907445 (+ 96us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.907549 (+ 104us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.907670 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"spinlock_wait_cycles":1280,"thread_start_us":87,"threads_started":1}
I20251214 03:02:53.908259 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.886202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.1.157.253:44661)
1214 03:02:53.886396 (+ 194us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.886400 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.886831 (+ 431us) server_negotiation.cc:408] Connection header received
1214 03:02:53.886983 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.886986 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.887032 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.887088 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.901889 (+ 14801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.902572 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.903613 (+ 1041us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.903788 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.904489 (+ 701us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.904510 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.904513 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.904539 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.906335 (+ 1796us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.906767 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.906770 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.906772 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.906817 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.907053 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.907056 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.907058 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.907223 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.907306 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.908048 (+ 742us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.908153 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20251214 03:02:53.913887 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.886427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.1.157.252:43123)
1214 03:02:53.886535 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.886540 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.886692 (+ 152us) server_negotiation.cc:408] Connection header received
1214 03:02:53.886850 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.886854 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.886895 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.886958 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.906376 (+ 19418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.907391 (+ 1015us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.909465 (+ 2074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.909637 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.910638 (+ 1001us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.910654 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.910657 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.910676 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.912387 (+ 1711us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.912804 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.912807 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.912809 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.912853 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.913063 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.913066 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.913068 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.913252 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.913334 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.913641 (+ 307us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.913777 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":28}
I20251214 03:02:53.913875 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.886130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.0.0.1:58834)
1214 03:02:53.886624 (+ 494us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.886637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.886754 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.887052 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.887055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.901429 (+ 14374us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.904361 (+ 2932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.904371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.907654 (+ 3283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.907656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.909342 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.909352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.909453 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.910108 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.910124 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.910526 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.912505 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.912508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.912511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.912706 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.912938 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.912941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.912943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.912986 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.913456 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.913457 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.913515 (+ 58us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.913588 (+ 73us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.913709 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":106,"threads_started":1}
I20251214 03:02:53.914705 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.886163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.1.157.254:42577)
1214 03:02:53.886292 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.886297 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.887010 (+ 713us) server_negotiation.cc:408] Connection header received
1214 03:02:53.887173 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.887177 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.887238 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.887290 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.902385 (+ 15095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.903124 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.908308 (+ 5184us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.908474 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.911230 (+ 2756us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.911246 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.911250 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.911275 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.912831 (+ 1556us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.913159 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.913163 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.913165 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.913244 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.913452 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.913455 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.913457 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.913618 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.913702 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.914451 (+ 749us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.914564 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251214 03:02:53.915150 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.886533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56134)
1214 03:02:53.886945 (+ 412us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.886956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.887064 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.887428 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.887431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.901965 (+ 14534us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.902239 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.902247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.903263 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.903266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.907537 (+ 4271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.907543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.908682 (+ 1139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.909995 (+ 1313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.910011 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.911094 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.912912 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.912915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.912918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.913071 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.913328 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.913331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.913333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.913379 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.913744 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.913747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.913825 (+ 78us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.914868 (+ 1043us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.914991 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":68,"threads_started":1}
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681372, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.254@KRBTEST.COM for oryx/127.1.157.252@KRBTEST.COM
I20251214 03:02:53.925413 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.918550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.0.0.1:58840)
1214 03:02:53.918712 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.918725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.918803 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.919032 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.919035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.919207 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.919448 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.919455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.920021 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.920025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.920564 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.920571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.920670 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.921072 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.921087 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.922466 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.924336 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.924338 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.924340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.924492 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.924691 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.924693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.924694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.924729 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.925021 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.925023 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.925089 (+ 66us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.925158 (+ 69us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.925306 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20251214 03:02:53.925616 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.918605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.1.157.252:43123)
1214 03:02:53.918709 (+ 104us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.918713 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.918776 (+ 63us) server_negotiation.cc:408] Connection header received
1214 03:02:53.918900 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.918903 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.918946 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.919006 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.919540 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.919942 (+ 402us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.920679 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.920848 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.922567 (+ 1719us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.922583 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.922587 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.922610 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.924241 (+ 1631us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.924572 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.924576 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.924578 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.924622 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.924789 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.924792 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.924794 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.924947 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.925030 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.925261 (+ 231us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.925361 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251214 03:02:53.926332 8416 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.1.157.252:43123: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416"
member_type: NON_VOTER
last_known_addr {
host: "127.1.157.252"
port: 43123
}
attrs {
promote: true
}
}
dest_uuid: "fc546e7566b6428ca27448a8b619ad2b"
cas_config_opid_index: 5
I20251214 03:02:53.926811 8416 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } }
I20251214 03:02:53.927703 8985 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:53.928120 9119 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:53.928972 8985 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.1.157.252:43123
I20251214 03:02:53.929124 8985 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:53.929207 9119 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 FOLLOWER]: Refusing update from remote peer fc546e7566b6428ca27448a8b619ad2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251214 03:02:53.929260 8985 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:53.929404 9037 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 LEARNER]: Refusing update from remote peer fc546e7566b6428ca27448a8b619ad2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251214 03:02:53.929837 8985 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 03:02:53.929970 9005 master.cc:561] Master@127.1.157.252:43123 shutting down...
I20251214 03:02:53.930096 8984 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 03:02:53.932189 8401 consensus_peers.cc:597] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b -> Peer e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252:43123): Couldn't send request to peer e930f3dd0c0a498d913e4239a7b82416. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251214 03:02:53.932862 9037 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:53.934962 9066 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc546e7566b6428ca27448a8b619ad2b. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:53.935066 9066 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: This master's current role is: LEARNER
I20251214 03:02:53.935096 9005 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:53.935148 9066 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: NON_VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: true } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:53.935200 9066 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: This master's current role is: LEARNER
I20251214 03:02:53.935314 9005 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:53.935384 9005 tablet_replica.cc:333] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: stopping tablet replica
I20251214 03:02:53.935424 8984 raft_consensus.cc:1064] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: attempting to promote NON_VOTER d5053326dfe84c4a97d163a30e8c4d60 to VOTER
I20251214 03:02:53.935817 8984 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } }
W20251214 03:02:53.936353 8416 master.cc:679] Remote error: unable to get registration information for peer e930f3dd0c0a498d913e4239a7b82416: Service unavailable: service kudu.master.MasterService not registered on Master
W20251214 03:02:53.937593 8401 consensus_peers.cc:597] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b -> Peer e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252:43123): Couldn't send request to peer e930f3dd0c0a498d913e4239a7b82416. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251214 03:02:53.938511 9037 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 LEARNER]: Refusing update from remote peer fc546e7566b6428ca27448a8b619ad2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20251214 03:02:53.938884 8985 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 03:02:53.940546 9175 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d5053326dfe84c4a97d163a30e8c4d60 (127.1.157.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:53.940917 9037 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d5053326dfe84c4a97d163a30e8c4d60 (127.1.157.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:53.941973 9174 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:53.942126 9174 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 03:02:53.943131 9175 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:53.943110 8985 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:53.943221 9175 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:53.943230 8985 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:53.950543 9005 master.cc:583] Master@127.1.157.252:43123 shutdown complete.
I20251214 03:02:53.950598 9005 master_runner.cc:305] Clearing existing system tablet
I20251214 03:02:53.950774 9005 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:53.950811 9005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:53.951802 9005 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:53.952538 9177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.952718 9005 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:53.952822 9005 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251214 03:02:53.952859 9005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal
uuid: "e930f3dd0c0a498d913e4239a7b82416"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "c08455fd0ede36a7ececfe1433dcc676"
server_key_iv: "a99802d0d52db509cbbbb65697e399ef"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.954413 9005 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:53.956598 9005 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251214 03:02:53.956686 9005 log.cc:1199] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20251214 03:02:53.956903 9005 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: Deleting consensus metadata
I20251214 03:02:53.957060 9005 master_runner.cc:315] Copying system tablet from 127.1.157.254:42577
I20251214 03:02:53.958271 9005 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: tablet copy: Beginning tablet copy session from remote peer at address 127.1.157.254:42577
I20251214 03:02:53.961094 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.960848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.0.0.1:58842)
1214 03:02:53.960961 (+ 113us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.961020 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.252:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20251214 03:02:53.961320 8416 master.cc:679] Network error: unable to get registration information for peer e930f3dd0c0a498d913e4239a7b82416: Client connection negotiation failed: client connection to 127.1.157.252:43123: connect: Connection refused (error 111)
I20251214 03:02:53.965525 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.958889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.1.157.254:42577)
1214 03:02:53.959098 (+ 209us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:53.959103 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:53.959174 (+ 71us) server_negotiation.cc:408] Connection header received
1214 03:02:53.959300 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:53.959303 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:53.959350 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:53.959414 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:53.960159 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.960960 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.961857 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.962007 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.962880 (+ 873us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:53.962894 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:53.962896 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:53.962917 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:53.964108 (+ 1191us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.964496 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.964500 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.964502 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.964548 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:53.964765 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:53.964768 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:53.964769 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:53.964940 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:53.965019 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:53.965320 (+ 301us) server_negotiation.cc:300] Negotiation successful
1214 03:02:53.965409 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251214 03:02:53.965559 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.958787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56142)
1214 03:02:53.959087 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.959101 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:53.959205 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:53.959437 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:53.959440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:53.959629 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:53.960040 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.960049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.961095 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:53.961099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:53.961742 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:53.961752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:53.961867 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:53.962372 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:53.962387 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:53.962760 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:53.964204 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.964208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.964210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.964409 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.964626 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:53.964629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:53.964631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:53.964681 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:53.965014 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:53.965017 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:53.965107 (+ 90us) client_negotiation.cc:769] Sending connection context
1214 03:02:53.965252 (+ 145us) client_negotiation.cc:240] Negotiation successful
1214 03:02:53.965382 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20251214 03:02:53.965926 8436 tablet_copy_service.cc:140] P fc546e7566b6428ca27448a8b619ad2b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e930f3dd0c0a498d913e4239a7b82416 ({username='oryx', principal='oryx/127.1.157.252@KRBTEST.COM'} at 127.0.0.1:56142)
I20251214 03:02:53.966010 8436 tablet_copy_service.cc:161] P fc546e7566b6428ca27448a8b619ad2b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e930f3dd0c0a498d913e4239a7b82416 at {username='oryx', principal='oryx/127.1.157.252@KRBTEST.COM'} at 127.0.0.1:56142: session id = e930f3dd0c0a498d913e4239a7b82416-00000000000000000000000000000000
I20251214 03:02:53.966737 8436 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 03:02:53.967803 9005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251214 03:02:53.969408 9005 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: tablet copy: Starting download of 0 data blocks...
I20251214 03:02:53.969555 9005 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: tablet copy: Starting download of 1 WAL segments...
I20251214 03:02:53.970845 9005 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 03:02:53.972139 8436 tablet_copy_service.cc:342] P fc546e7566b6428ca27448a8b619ad2b: Request end of tablet copy session e930f3dd0c0a498d913e4239a7b82416-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.157.252@KRBTEST.COM'} at 127.0.0.1:56142
I20251214 03:02:53.972203 8436 tablet_copy_service.cc:434] P fc546e7566b6428ca27448a8b619ad2b: ending tablet copy session e930f3dd0c0a498d913e4239a7b82416-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e930f3dd0c0a498d913e4239a7b82416
W20251214 03:02:53.975808 9005 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251214 03:02:53.977685 9005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:53.979148 9185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:53.979209 9184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:53.979362 9187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:53.981316 9005 server_base.cc:1047] running on GCE node
I20251214 03:02:53.981451 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:53.981693 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:53.982818 9005 hybrid_clock.cc:648] HybridClock initialized: now 1765681373982805 us; error 29 us; skew 500 ppm
Dec 14 03:02:53 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:53.984331 9005 init.cc:377] Logged in from keytab as oryx/127.1.157.252@KRBTEST.COM (short username oryx)
I20251214 03:02:53.984781 9005 webserver.cc:492] Webserver started at http://127.1.157.252:37395/ using document root <none> and password file <none>
I20251214 03:02:53.984956 9005 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:53.984995 9005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:53.986218 9005 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251214 03:02:53.987126 9194 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:53.987373 9005 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251214 03:02:53.987445 9005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal
uuid: "e930f3dd0c0a498d913e4239a7b82416"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "c08455fd0ede36a7ececfe1433dcc676"
server_key_iv: "a99802d0d52db509cbbbb65697e399ef"
server_key_version: "encryptionkey@0"
I20251214 03:02:53.987521 9005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:53.996964 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:53.996707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.0.0.1:58848)
1214 03:02:53.996835 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:53.996888 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.252:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20251214 03:02:53.997205 8416 master.cc:679] Network error: unable to get registration information for peer e930f3dd0c0a498d913e4239a7b82416: Client connection negotiation failed: client connection to 127.1.157.252:43123: connect: Connection refused (error 111)
I20251214 03:02:54.015898 9005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:54.017871 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:54.022385 9005 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.252:43123
I20251214 03:02:54.022516 9246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.252:43123 every 8 connection(s)
I20251214 03:02:54.023010 9005 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/master-2/data/info.pb
I20251214 03:02:54.024461 9247 sys_catalog.cc:263] Verifying existing consensus state
I20251214 03:02:54.025269 9247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: Bootstrap starting.
I20251214 03:02:54.033928 9247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 03:02:54.034173 9247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: Bootstrap complete.
I20251214 03:02:54.034744 9247 raft_consensus.cc:359] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } }
I20251214 03:02:54.034847 9247 raft_consensus.cc:740] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e930f3dd0c0a498d913e4239a7b82416, State: Initialized, Role: LEARNER
I20251214 03:02:54.034947 9247 consensus_queue.cc:260] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } }
I20251214 03:02:54.035271 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:54.035352 9247 sys_catalog.cc:565] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:02:54.035419 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: This master's current role is: LEARNER
I20251214 03:02:54.037871 9261 catalog_manager.cc:1269] Loaded cluster ID: 4fcd4b0673a74ba5a0393d9fa9c2a01e
I20251214 03:02:54.037917 9261 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: loading cluster ID for follower catalog manager: success
I20251214 03:02:54.039577 9261 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: acquiring CA information for follower catalog manager: success
I20251214 03:02:54.040858 9261 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251214 03:02:54.068738 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:54.064894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.0.0.1:58860)
1214 03:02:54.065074 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:54.065090 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:54.065202 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:54.065618 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:54.065621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:54.065641 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:54.065880 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:54.065887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:54.066832 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:54.066836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:54.067788 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:54.067801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:54.068244 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:54.068290 (+ 46us) client_negotiation.cc:769] Sending connection context
1214 03:02:54.068381 (+ 91us) client_negotiation.cc:240] Negotiation successful
1214 03:02:54.068517 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20251214 03:02:54.068913 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:54.065054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.1.157.252:43123)
1214 03:02:54.065343 (+ 289us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:54.065347 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:54.065361 (+ 14us) server_negotiation.cc:408] Connection header received
1214 03:02:54.065422 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:54.065425 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:54.065477 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:54.065585 (+ 108us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:54.066014 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:54.066722 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:54.068175 (+ 1453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:54.068555 (+ 380us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:54.068667 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:54.068723 (+ 56us) server_negotiation.cc:300] Negotiation successful
1214 03:02:54.068775 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":139,"threads_started":1}
I20251214 03:02:54.074371 8745 heartbeater.cc:499] Master 127.1.157.254:42577 was elected leader, sending a full tablet report...
I20251214 03:02:54.288056 8881 heartbeater.cc:499] Master 127.1.157.254:42577 was elected leader, sending a full tablet report...
I20251214 03:02:54.454454 9221 raft_consensus.cc:1217] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251214 03:02:54.454876 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc546e7566b6428ca27448a8b619ad2b. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:54.454994 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: This master's current role is: LEARNER
I20251214 03:02:54.998330 8985 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: NON_VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: true } } }
I20251214 03:02:54.998512 8985 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:54.999378 9264 raft_consensus.cc:1064] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: attempting to promote NON_VOTER e930f3dd0c0a498d913e4239a7b82416 to VOTER
I20251214 03:02:54.999733 9264 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } }
I20251214 03:02:55.000881 9221 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 LEARNER]: Refusing update from remote peer fc546e7566b6428ca27448a8b619ad2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251214 03:02:55.000906 9037 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Refusing update from remote peer fc546e7566b6428ca27448a8b619ad2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251214 03:02:55.001335 9263 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 03:02:55.001538 8985 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 03:02:55.003327 8985 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:02:55.003793 9037 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:02:55.004801 8985 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:02:55.004895 8985 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:55.004629 9221 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:02:55.005026 8985 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:02:55.005093 8985 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:55.005244 8985 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:02:55.005318 8985 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b [sys.catalog]: This master's current role is: LEADER
I20251214 03:02:55.005750 9269 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:02:55.005909 9269 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 03:02:55.006237 9270 sys_catalog.cc:455] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc546e7566b6428ca27448a8b619ad2b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:02:55.006333 9270 sys_catalog.cc:458] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 03:02:55.989979 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:55.983429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.1.157.253:44661)
1214 03:02:55.983647 (+ 218us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:55.983651 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:55.983663 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:02:55.983762 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:55.983765 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:55.983803 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:55.983872 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:55.984671 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.985200 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:55.985850 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.986016 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:55.987018 (+ 1002us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:55.987034 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:55.987036 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:55.987057 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:55.988600 (+ 1543us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:55.988944 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:55.988947 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:55.988948 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:55.988985 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:55.989213 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:55.989217 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:55.989219 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:55.989369 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:55.989503 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:55.989703 (+ 200us) server_negotiation.cc:300] Negotiation successful
1214 03:02:55.989820 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":74,"threads_started":1}
I20251214 03:02:55.992904 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:55.990693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56150)
1214 03:02:55.990901 (+ 208us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:55.990912 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:55.990980 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:55.991208 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:55.991210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:55.991221 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:55.991397 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:55.991402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.992043 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:55.992046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:55.992624 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:55.992630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.992713 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:55.992726 (+ 13us) client_negotiation.cc:769] Sending connection context
1214 03:02:55.992758 (+ 32us) client_negotiation.cc:240] Negotiation successful
1214 03:02:55.992803 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20251214 03:02:55.993341 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:55.990819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.1.157.254:42577)
1214 03:02:55.991029 (+ 210us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:55.991033 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:55.991045 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:02:55.991083 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:55.991085 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:55.991126 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:55.991200 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:55.991470 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.991975 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:55.992708 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.993062 (+ 354us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:55.993103 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:55.993150 (+ 47us) server_negotiation.cc:300] Negotiation successful
1214 03:02:55.993244 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":57,"threads_started":1}
I20251214 03:02:55.996474 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:55.994353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.0.0.1:41958)
1214 03:02:55.994438 (+ 85us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:55.994448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:55.994520 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:55.994728 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:55.994731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:55.994745 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:55.994894 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:55.994899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.995675 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:55.995678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:55.996203 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:55.996210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.996299 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:55.996312 (+ 13us) client_negotiation.cc:769] Sending connection context
1214 03:02:55.996343 (+ 31us) client_negotiation.cc:240] Negotiation successful
1214 03:02:55.996382 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20251214 03:02:55.996807 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:55.994451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.1.157.253:44661)
1214 03:02:55.994559 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:55.994562 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:55.994573 (+ 11us) server_negotiation.cc:408] Connection header received
1214 03:02:55.994612 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:55.994614 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:55.994652 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:55.994722 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:55.995024 (+ 302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.995587 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:55.996295 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.996622 (+ 327us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:55.996653 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:55.996693 (+ 40us) server_negotiation.cc:300] Negotiation successful
1214 03:02:55.996733 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251214 03:02:55.999855 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:55.997784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.0.0.1:58874)
1214 03:02:55.997891 (+ 107us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:55.997903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:55.997960 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:55.998300 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:55.998302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:55.998314 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:55.998455 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:55.998460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.999084 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:55.999086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:55.999616 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:55.999621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.999699 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:55.999711 (+ 12us) client_negotiation.cc:769] Sending connection context
1214 03:02:55.999741 (+ 30us) client_negotiation.cc:240] Negotiation successful
1214 03:02:55.999779 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20251214 03:02:56.000272 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:55.997925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.1.157.252:43123)
1214 03:02:55.998126 (+ 201us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:55.998129 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:55.998141 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:02:55.998176 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:55.998178 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:55.998215 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:55.998282 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:55.998536 (+ 254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:55.999011 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:55.999695 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.000063 (+ 368us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.000094 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.000132 (+ 38us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.000179 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20251214 03:02:56.007846 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.001826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.1.157.252:43123)
1214 03:02:56.001935 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.001939 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.001951 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:02:56.002013 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.002016 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.002053 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.002110 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.002793 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.003231 (+ 438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.003828 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.004018 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.004953 (+ 935us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.004969 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.004971 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.004990 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.006622 (+ 1632us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.007004 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.007008 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.007010 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.007064 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.007251 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.007253 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.007254 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.007396 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.007501 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.007639 (+ 138us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.007733 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251214 03:02:56.010876 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.008582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56156)
1214 03:02:56.008782 (+ 200us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.008793 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.008863 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.009075 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.009078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.009095 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:56.009315 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.009322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.010014 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.010016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.010596 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.010603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.010688 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.010702 (+ 14us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.010733 (+ 31us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.010777 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":55,"threads_started":1}
I20251214 03:02:56.011332 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.008709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.1.157.254:42577)
1214 03:02:56.008862 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.008866 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.008881 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:02:56.008924 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.008926 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.008966 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.009035 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:56.009428 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.009925 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.010705 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.011083 (+ 378us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.011123 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.011164 (+ 41us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.011243 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251214 03:02:56.014609 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.012483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.0.0.1:41966)
1214 03:02:56.012607 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.012622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.012686 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.012883 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.012885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.012897 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:56.013048 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.013052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.013722 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.013725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.014312 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.014320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.014422 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.014436 (+ 14us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.014475 (+ 39us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.014522 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20251214 03:02:56.015013 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.012592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.1.157.253:44661)
1214 03:02:56.012715 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.012719 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.012731 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:02:56.012769 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.012771 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.012809 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.012871 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:56.013127 (+ 256us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.013637 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.014442 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.014803 (+ 361us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.014839 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.014882 (+ 43us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.014930 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251214 03:02:56.018057 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.016023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.0.0.1:58886)
1214 03:02:56.016119 (+ 96us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.016132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.016201 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.016398 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.016401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.016415 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:56.016552 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.016557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.017166 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.017196 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.017736 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.017743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.017838 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.017855 (+ 17us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.017901 (+ 46us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.017950 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20251214 03:02:56.018417 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.016060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.1.157.252:43123)
1214 03:02:56.016152 (+ 92us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.016156 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.016168 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:02:56.016293 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.016296 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.016332 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.016388 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:56.016638 (+ 250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.017079 (+ 441us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.017837 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.018222 (+ 385us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.018250 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.018300 (+ 50us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.018342 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37,"spinlock_wait_cycles":896}
I20251214 03:02:56.019574 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 8480
I20251214 03:02:56.024641 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:36175
--local_ip_for_outbound_sockets=127.1.157.193
--tserver_master_addrs=127.1.157.254:42577
--webserver_port=42615
--webserver_interface=127.1.157.193
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.157.254:42577,127.1.157.253:44661
--tserver_master_addrs=127.1.157.254:42577,127.1.157.253:44661,127.1.157.252:43123 with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:02:56.131960 9277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:56.132207 9277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:56.132305 9277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:56.135615 9277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:56.135673 9277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:56.135752 9277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:02:56.139879 9277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:36175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=42615
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:42577,127.1.157.253:44661,127.1.157.252:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:56.140848 9277 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:56.141638 9277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:56.148205 9283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:56.148270 9282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:56.148208 9285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:56.148551 9277 server_base.cc:1047] running on GCE node
I20251214 03:02:56.148993 9277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:56.149606 9277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:56.150775 9277 hybrid_clock.cc:648] HybridClock initialized: now 1765681376150760 us; error 30 us; skew 500 ppm
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:56.153707 9277 init.cc:377] Logged in from keytab as oryx/127.1.157.193@KRBTEST.COM (short username oryx)
I20251214 03:02:56.154915 9277 webserver.cc:492] Webserver started at http://127.1.157.193:42615/ using document root <none> and password file <none>
I20251214 03:02:56.155470 9277 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:56.155535 9277 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:56.159022 9277 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251214 03:02:56.160851 9292 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:56.161819 9277 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:56.161922 9277 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "74d4903d11c04204a94556ce1adb4088"
format_stamp: "Formatted at 2025-12-14 03:02:52 on dist-test-slave-k7z7"
server_key: "fbaee491e9bd6a1e758b02cd552cca4c"
server_key_iv: "4d582c79a8bfb28847492a0fead724b1"
server_key_version: "encryptionkey@0"
I20251214 03:02:56.162328 9277 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:56.203312 9277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:56.206286 9277 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:56.206478 9277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:56.207072 9277 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:56.208026 9277 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:56.208073 9277 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:56.208115 9277 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:56.208144 9277 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:56.217651 9277 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:36175
I20251214 03:02:56.217712 9405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:36175 every 8 connection(s)
I20251214 03:02:56.218622 9277 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:02:56.220901 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 9277
I20251214 03:02:56.221096 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 8616
I20251214 03:02:56.229524 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:43857
--local_ip_for_outbound_sockets=127.1.157.194
--tserver_master_addrs=127.1.157.254:42577
--webserver_port=44773
--webserver_interface=127.1.157.194
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.157.254:42577,127.1.157.253:44661
--tserver_master_addrs=127.1.157.254:42577,127.1.157.253:44661,127.1.157.252:43123 with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.253@KRBTEST.COM
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.252@KRBTEST.COM
I20251214 03:02:56.235616 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.220344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:41819 (local address 127.1.157.253:44661)
1214 03:02:56.220486 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.220491 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.225491 (+ 5000us) server_negotiation.cc:408] Connection header received
1214 03:02:56.225543 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.225545 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.225593 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.225668 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.226365 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.227132 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.227959 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.228135 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.231116 (+ 2981us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.231140 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.231142 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.231170 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.233002 (+ 1832us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.233780 (+ 778us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.233784 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.233786 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.233846 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.234206 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.234210 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.234212 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.234425 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.234523 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.235278 (+ 755us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.235425 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251214 03:02:56.235672 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.221355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:46139 (local address 127.1.157.252:43123)
1214 03:02:56.221492 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.221497 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.224421 (+ 2924us) server_negotiation.cc:408] Connection header received
1214 03:02:56.224513 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.224516 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.224573 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.224658 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.225951 (+ 1293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.226755 (+ 804us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.227690 (+ 935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.227882 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.232155 (+ 4273us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.232177 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.232181 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.232212 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.233933 (+ 1721us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.234465 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.234469 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.234471 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.234530 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.234771 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.234775 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.234777 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.234967 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.235047 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.235383 (+ 336us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.235516 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251214 03:02:56.235673 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.220345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:33031 (local address 127.1.157.254:42577)
1214 03:02:56.220493 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.220498 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.222717 (+ 2219us) server_negotiation.cc:408] Connection header received
1214 03:02:56.223821 (+ 1104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.223824 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.223957 (+ 133us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.224061 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.225951 (+ 1890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.226755 (+ 804us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.227690 (+ 935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.227893 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.231798 (+ 3905us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.231823 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.231827 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.231862 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.234110 (+ 2248us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.234544 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.234548 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.234550 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.234607 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.234834 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.234837 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.234839 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.235001 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.235070 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.235388 (+ 318us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.235516 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251214 03:02:56.237082 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.220658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.193:33031)
1214 03:02:56.222532 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.222584 (+ 52us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.223580 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.224281 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.224291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.224904 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:56.225766 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.225783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.226915 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.226919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.227554 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.227563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.228132 (+ 569us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.228941 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:56.228954 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:56.231654 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:56.234221 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.234225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.234227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.234444 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.234696 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.234699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.234701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.234744 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.235115 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:56.235118 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:56.235199 (+ 81us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.235820 (+ 621us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.236221 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1653,"mutex_wait_us":118,"spinlock_wait_cycles":13056,"thread_start_us":127,"threads_started":1}
I20251214 03:02:56.237087 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.220909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.1.157.193:46139)
1214 03:02:56.224326 (+ 3417us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.224337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.224420 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.224771 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.224775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.225602 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:56.225824 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.225830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.226915 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.226919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.227554 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.227563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.228132 (+ 569us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.228888 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:56.228925 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:56.232003 (+ 3078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:56.234110 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.234114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.234116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.234356 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.234626 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.234629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.234631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.234680 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.235103 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:56.235106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:56.235199 (+ 93us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.235856 (+ 657us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.236139 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3361,"spinlock_wait_cycles":19328,"thread_start_us":90,"threads_started":1}
I20251214 03:02:56.237087 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.220658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.1.157.193:41819)
1214 03:02:56.224050 (+ 3392us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.224063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.224149 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.225711 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.225714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.225907 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:56.226217 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.226226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.227282 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.227286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.227830 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.227840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.228257 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.229004 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:56.229021 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:56.230900 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:56.233261 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.233267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.233282 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.233654 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.233957 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.233960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.233963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.234095 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.234598 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:56.234606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:56.234954 (+ 348us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.235821 (+ 867us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.236320 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3292,"mutex_wait_us":247,"spinlock_wait_cycles":1280,"thread_start_us":116,"threads_started":1}
I20251214 03:02:56.238543 9406 heartbeater.cc:344] Connected to a master server at 127.1.157.252:43123
I20251214 03:02:56.238561 9408 heartbeater.cc:344] Connected to a master server at 127.1.157.253:44661
I20251214 03:02:56.238811 9406 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:56.238812 9408 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:56.239423 9408 heartbeater.cc:507] Master 127.1.157.253:44661 requested a full tablet report, sending...
I20251214 03:02:56.239454 9407 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42577
I20251214 03:02:56.239574 9407 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:56.239732 9407 heartbeater.cc:507] Master 127.1.157.254:42577 requested a full tablet report, sending...
I20251214 03:02:56.239421 9406 heartbeater.cc:507] Master 127.1.157.252:43123 requested a full tablet report, sending...
I20251214 03:02:56.240684 8416 ts_manager.cc:194] Re-registered known tserver with Master: 74d4903d11c04204a94556ce1adb4088 (127.1.157.193:36175)
I20251214 03:02:56.241102 9211 ts_manager.cc:194] Registered new tserver with Master: 74d4903d11c04204a94556ce1adb4088 (127.1.157.193:36175)
I20251214 03:02:56.241309 8416 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:33031
I20251214 03:02:56.242209 9027 ts_manager.cc:194] Registered new tserver with Master: 74d4903d11c04204a94556ce1adb4088 (127.1.157.193:36175)
W20251214 03:02:56.343526 9413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:56.343756 9413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:56.343816 9413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:56.347206 9413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:56.347272 9413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:56.347373 9413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:02:56.351529 9413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:43857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=44773
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:42577,127.1.157.253:44661,127.1.157.252:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:56.352542 9413 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:56.353348 9413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:56.359607 9419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:56.359629 9418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:56.359608 9421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:56.359872 9413 server_base.cc:1047] running on GCE node
I20251214 03:02:56.360224 9413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:56.360795 9413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:56.361948 9413 hybrid_clock.cc:648] HybridClock initialized: now 1765681376361929 us; error 29 us; skew 500 ppm
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:56.364698 9413 init.cc:377] Logged in from keytab as oryx/127.1.157.194@KRBTEST.COM (short username oryx)
I20251214 03:02:56.365808 9413 webserver.cc:492] Webserver started at http://127.1.157.194:44773/ using document root <none> and password file <none>
I20251214 03:02:56.366321 9413 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:56.366382 9413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:56.369752 9413 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251214 03:02:56.371508 9428 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:56.372463 9413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:02:56.372570 9413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "af3d55d43111412b82f233739185d287"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "2513802fbbc6010cb28de5cd6d0d8f0c"
server_key_iv: "ca161b52d70a63cb21c1a34822ce60b3"
server_key_version: "encryptionkey@0"
I20251214 03:02:56.372948 9413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:56.397709 9413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:56.400204 9413 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:56.400364 9413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:56.400894 9413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:56.401778 9413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:56.401826 9413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:56.401878 9413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:56.401902 9413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:56.410542 9413 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:43857
I20251214 03:02:56.410557 9541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:43857 every 8 connection(s)
I20251214 03:02:56.411525 9413 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:02:56.416234 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 9413
I20251214 03:02:56.416420 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 8752
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.253@KRBTEST.COM
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.252@KRBTEST.COM
I20251214 03:02:56.423421 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:41383
--local_ip_for_outbound_sockets=127.1.157.195
--tserver_master_addrs=127.1.157.254:42577
--webserver_port=46097
--webserver_interface=127.1.157.195
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.157.254:42577,127.1.157.253:44661
--tserver_master_addrs=127.1.157.254:42577,127.1.157.253:44661,127.1.157.252:43123 with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251214 03:02:56.426314 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.413239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:37779 (local address 127.1.157.252:43123)
1214 03:02:56.413405 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.413409 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.414307 (+ 898us) server_negotiation.cc:408] Connection header received
1214 03:02:56.415535 (+ 1228us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.415539 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.415588 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.415639 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.417057 (+ 1418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.417865 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.418741 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.418926 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.422840 (+ 3914us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.422858 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.422861 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.422886 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.424584 (+ 1698us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.425016 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.425020 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.425022 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.425066 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.425336 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.425339 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.425341 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.425503 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.425561 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.426036 (+ 475us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.426157 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251214 03:02:56.426697 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.413813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:56017 (local address 127.1.157.253:44661)
1214 03:02:56.413934 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.413938 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.414634 (+ 696us) server_negotiation.cc:408] Connection header received
1214 03:02:56.415351 (+ 717us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.415355 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.415409 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.415485 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.417056 (+ 1571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.417865 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.418741 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.418926 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.421718 (+ 2792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.421737 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.421740 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.421766 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.424140 (+ 2374us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.424794 (+ 654us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.424798 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.424799 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.424849 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.425196 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.425200 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.425202 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.425391 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.425478 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.426471 (+ 993us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.426580 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251214 03:02:56.427013 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.413589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.1.157.194:37779)
1214 03:02:56.414167 (+ 578us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.414208 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.415428 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.415758 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.415761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.416080 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:56.416900 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.416915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.418009 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.418013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.418617 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.418625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.419146 (+ 521us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.419902 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:56.419915 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:56.422719 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:56.424709 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.424713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.424715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.424918 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.425155 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.425157 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.425159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.425231 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.425615 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:56.425618 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:56.425789 (+ 171us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.425966 (+ 177us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.426302 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":91,"spinlock_wait_cycles":6784,"thread_start_us":113,"threads_started":1}
I20251214 03:02:56.427026 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.413668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.1.157.194:56017)
1214 03:02:56.414554 (+ 886us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.414567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.415162 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.415637 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.415646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.416533 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:56.416900 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.416915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.418009 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.418013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.418619 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.418625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.419146 (+ 521us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.419798 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:56.419831 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:56.421582 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:56.424237 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.424244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.424257 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.424686 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.424946 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.424950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.424952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.425073 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.425533 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:56.425540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:56.425791 (+ 251us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.425965 (+ 174us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.426211 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":66,"spinlock_wait_cycles":16512,"thread_start_us":94,"threads_started":1}
I20251214 03:02:56.427560 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.413589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.194:51839)
1214 03:02:56.414440 (+ 851us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.414450 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.415151 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.415636 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.415644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.416087 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:56.417094 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.417103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.419336 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.419340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.419886 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.419895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.419995 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.420713 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:56.420728 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:56.422394 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:56.424237 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.424244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.424263 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.424686 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.424946 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.424950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.424952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.425073 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.427061 (+ 1988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:56.427065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:56.427149 (+ 84us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.427263 (+ 114us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.427382 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":22,"spinlock_wait_cycles":65920,"thread_start_us":92,"threads_started":1}
I20251214 03:02:56.427629 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.413240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:51839 (local address 127.1.157.254:42577)
1214 03:02:56.413405 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.413409 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.414527 (+ 1118us) server_negotiation.cc:408] Connection header received
1214 03:02:56.415339 (+ 812us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.415343 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.415404 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.415475 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.418443 (+ 2968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.419193 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.419995 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.420475 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.422541 (+ 2066us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.422562 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.422565 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.422593 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.424103 (+ 1510us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.424798 (+ 695us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.424801 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.424803 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.424849 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.425217 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.425220 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.425222 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.425405 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.425501 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.427347 (+ 1846us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.427478 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20251214 03:02:56.428265 9542 heartbeater.cc:344] Connected to a master server at 127.1.157.252:43123
I20251214 03:02:56.428510 9542 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:56.429021 9542 heartbeater.cc:507] Master 127.1.157.252:43123 requested a full tablet report, sending...
I20251214 03:02:56.429266 9544 heartbeater.cc:344] Connected to a master server at 127.1.157.253:44661
I20251214 03:02:56.429354 9544 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:56.429549 9544 heartbeater.cc:507] Master 127.1.157.253:44661 requested a full tablet report, sending...
I20251214 03:02:56.429839 9543 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42577
I20251214 03:02:56.429906 9543 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:56.430083 9543 heartbeater.cc:507] Master 127.1.157.254:42577 requested a full tablet report, sending...
I20251214 03:02:56.430117 9211 ts_manager.cc:194] Registered new tserver with Master: af3d55d43111412b82f233739185d287 (127.1.157.194:43857)
I20251214 03:02:56.430166 9027 ts_manager.cc:194] Registered new tserver with Master: af3d55d43111412b82f233739185d287 (127.1.157.194:43857)
I20251214 03:02:56.430851 8416 ts_manager.cc:194] Re-registered known tserver with Master: af3d55d43111412b82f233739185d287 (127.1.157.194:43857)
I20251214 03:02:56.431455 8416 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:51839
W20251214 03:02:56.532353 9549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:02:56.532567 9549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:02:56.532622 9549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:02:56.535790 9549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:02:56.535843 9549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:02:56.535919 9549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:02:56.539935 9549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:41383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=46097
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:42577,127.1.157.253:44661,127.1.157.252:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:02:56.540879 9549 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:02:56.541682 9549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:02:56.547783 9555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:56.547793 9557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:02:56.547791 9554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:02:56.548031 9549 server_base.cc:1047] running on GCE node
I20251214 03:02:56.548339 9549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:02:56.548848 9549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:02:56.549986 9549 hybrid_clock.cc:648] HybridClock initialized: now 1765681376549975 us; error 26 us; skew 500 ppm
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:02:56.552392 9549 init.cc:377] Logged in from keytab as oryx/127.1.157.195@KRBTEST.COM (short username oryx)
I20251214 03:02:56.553351 9549 webserver.cc:492] Webserver started at http://127.1.157.195:46097/ using document root <none> and password file <none>
I20251214 03:02:56.553861 9549 fs_manager.cc:362] Metadata directory not provided
I20251214 03:02:56.553925 9549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:02:56.557087 9549 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251214 03:02:56.558789 9564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:56.559715 9549 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:02:56.559809 9549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "7dd607b9ac1a4c779da1359d96d5b5f9"
format_stamp: "Formatted at 2025-12-14 03:02:53 on dist-test-slave-k7z7"
server_key: "247135edee0d7e4a75e80edeb661a519"
server_key_iv: "861bf1e8cf2a35248d9e3e7918746d5d"
server_key_version: "encryptionkey@0"
I20251214 03:02:56.560187 9549 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:02:56.580183 9549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:02:56.583063 9549 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:02:56.583227 9549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:02:56.583793 9549 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:02:56.584705 9549 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:02:56.584753 9549 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:56.584793 9549 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:02:56.584820 9549 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:02:56.594147 9549 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:41383
I20251214 03:02:56.594185 9677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:41383 every 8 connection(s)
I20251214 03:02:56.595157 9549 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:02:56.601389 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 9549
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.252@KRBTEST.COM
Dec 14 03:02:56 dist-test-slave-k7z7 krb5kdc[8368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681376, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.253@KRBTEST.COM
I20251214 03:02:56.609072 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.596877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:33879 (local address 127.1.157.253:44661)
1214 03:02:56.597048 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.597053 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.598243 (+ 1190us) server_negotiation.cc:408] Connection header received
1214 03:02:56.599044 (+ 801us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.599048 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.599093 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.599155 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.600267 (+ 1112us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.601032 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.602308 (+ 1276us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.602486 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.605504 (+ 3018us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.605522 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.605525 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.605552 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.607401 (+ 1849us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.607926 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.607929 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.607930 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.607979 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.608198 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.608201 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.608203 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.608350 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.608429 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.608813 (+ 384us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.608918 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251214 03:02:56.609128 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.597268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:49061 (local address 127.1.157.252:43123)
1214 03:02:56.597391 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.597395 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.598321 (+ 926us) server_negotiation.cc:408] Connection header received
1214 03:02:56.598793 (+ 472us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.598796 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.598842 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.598914 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.600362 (+ 1448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.600853 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.601646 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.601818 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.604888 (+ 3070us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.604905 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.604908 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.604933 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.607586 (+ 2653us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.608018 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.608021 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.608023 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.608068 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.608309 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.608312 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.608314 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.608478 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.608563 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.608888 (+ 325us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.608986 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251214 03:02:56.609651 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.597144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.195:51063)
1214 03:02:56.597627 (+ 483us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.597674 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.598576 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.599080 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.599091 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.599607 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:56.600155 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.600164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.601143 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.601146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.601659 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.601667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.601845 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.602571 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:56.602602 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:56.604404 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:56.606912 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.606918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.606930 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.607240 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.607536 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.607540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.607542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.607659 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.608064 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:56.608070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:56.608342 (+ 272us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.608564 (+ 222us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.608837 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"spinlock_wait_cycles":91776,"thread_start_us":166,"threads_started":1}
I20251214 03:02:56.610339 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.597258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.253:44661 (local address 127.1.157.195:33879)
1214 03:02:56.598154 (+ 896us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.598165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.598576 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.599184 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.599187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.599595 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:56.600133 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.600144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.601225 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.601228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.602152 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.602163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.602597 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.603388 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:56.603403 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:56.605390 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:56.607534 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.607538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.607540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.607833 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.608067 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.608071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.608073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.608120 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.608468 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:56.608471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:56.608559 (+ 88us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.608659 (+ 100us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.610176 (+ 1517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":1361,"spinlock_wait_cycles":3456,"thread_start_us":117,"threads_started":1}
I20251214 03:02:56.610384 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.597144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.252:43123 (local address 127.1.157.195:49061)
1214 03:02:56.598243 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.598256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.598594 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.599080 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.599091 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.600006 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:02:56.600261 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.600268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.600963 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.600966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.601520 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.601529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.601927 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.602667 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:02:56.602681 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:02:56.604762 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:02:56.607687 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.607690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.607692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.607914 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.608165 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:02:56.608168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:02:56.608170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:02:56.608219 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:02:56.608601 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:02:56.608604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:02:56.608676 (+ 72us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.608764 (+ 88us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.610249 (+ 1485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":2102,"thread_start_us":97,"threads_started":1}
I20251214 03:02:56.610823 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.597261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:51063 (local address 127.1.157.254:42577)
1214 03:02:56.597388 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.597392 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.597778 (+ 386us) server_negotiation.cc:408] Connection header received
1214 03:02:56.598766 (+ 988us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.598769 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.598825 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.598903 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.600275 (+ 1372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.601039 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.601757 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.601942 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.604669 (+ 2727us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.604686 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.604690 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.604721 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.606778 (+ 2057us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.607363 (+ 585us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.607367 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.607368 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.607430 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.607754 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.607757 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.607759 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.607935 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.608016 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.610556 (+ 2540us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.610685 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251214 03:02:56.611271 9680 heartbeater.cc:344] Connected to a master server at 127.1.157.253:44661
I20251214 03:02:56.611536 9679 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42577
I20251214 03:02:56.611541 9678 heartbeater.cc:344] Connected to a master server at 127.1.157.252:43123
I20251214 03:02:56.611537 9680 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:56.611647 9679 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:56.611647 9678 heartbeater.cc:461] Registering TS with master...
I20251214 03:02:56.612231 9679 heartbeater.cc:507] Master 127.1.157.254:42577 requested a full tablet report, sending...
I20251214 03:02:56.612231 9680 heartbeater.cc:507] Master 127.1.157.253:44661 requested a full tablet report, sending...
I20251214 03:02:56.612231 9678 heartbeater.cc:507] Master 127.1.157.252:43123 requested a full tablet report, sending...
I20251214 03:02:56.613349 9211 ts_manager.cc:194] Registered new tserver with Master: 7dd607b9ac1a4c779da1359d96d5b5f9 (127.1.157.195:41383)
I20251214 03:02:56.613377 9027 ts_manager.cc:194] Registered new tserver with Master: 7dd607b9ac1a4c779da1359d96d5b5f9 (127.1.157.195:41383)
I20251214 03:02:56.613386 8416 ts_manager.cc:194] Re-registered known tserver with Master: 7dd607b9ac1a4c779da1359d96d5b5f9 (127.1.157.195:41383)
I20251214 03:02:56.614059 8416 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:51063
I20251214 03:02:56.619340 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.603217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.1.157.253:44661)
1214 03:02:56.603849 (+ 632us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.603853 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.604833 (+ 980us) server_negotiation.cc:408] Connection header received
1214 03:02:56.605004 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.605007 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.605055 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.605114 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.607925 (+ 2811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.608632 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.613824 (+ 5192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.614008 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.615674 (+ 1666us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.615692 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.615695 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.615720 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.617690 (+ 1970us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.618146 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.618150 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.618152 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.618201 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.618496 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.618500 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.618501 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.618657 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.618756 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.619064 (+ 308us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.619192 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":71,"threads_started":1}
I20251214 03:02:56.620260 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.603841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.1.157.254:42577)
1214 03:02:56.604130 (+ 289us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.604135 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.605523 (+ 1388us) server_negotiation.cc:408] Connection header received
1214 03:02:56.605561 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.605564 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.605608 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.605684 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.609515 (+ 3831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.610052 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.614558 (+ 4506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.614757 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.616211 (+ 1454us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.616228 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.616231 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.616254 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.617957 (+ 1703us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.618337 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.618340 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.618341 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.618380 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.618801 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.618804 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.618805 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.618937 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.619050 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.619992 (+ 942us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.620112 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":64,"threads_started":1}
I20251214 03:02:56.620497 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.604208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.1.157.252:43123)
1214 03:02:56.604445 (+ 237us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.604449 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.610126 (+ 5677us) server_negotiation.cc:408] Connection header received
1214 03:02:56.610158 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.610160 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.610199 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.610278 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:56.611470 (+ 1192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.614132 (+ 2662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.615865 (+ 1733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.616047 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.616819 (+ 772us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:56.616839 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:56.616842 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:56.616868 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:56.618867 (+ 1999us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.619498 (+ 631us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.619501 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.619503 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.619551 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:56.619767 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:56.619769 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:56.619770 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:56.619894 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:56.620009 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.620236 (+ 227us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.620344 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":81,"threads_started":1}
I20251214 03:02:56.622526 8416 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:02:56.624677 8416 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:02:56.640765 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.637164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:36175 (local address 127.0.0.1:45366)
1214 03:02:56.637863 (+ 699us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.637878 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.637975 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.638504 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.638506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.638520 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:56.638722 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.638727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.639830 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.639833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.640458 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.640466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.640553 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.640572 (+ 19us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.640604 (+ 32us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.640648 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":133,"threads_started":1}
I20251214 03:02:56.641742 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.637351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.1.157.193:36175)
1214 03:02:56.637688 (+ 337us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.637695 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.638067 (+ 372us) server_negotiation.cc:408] Connection header received
1214 03:02:56.638152 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.638158 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.638370 (+ 212us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.638496 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:56.638846 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.639703 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.640596 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.641344 (+ 748us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.641461 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.641527 (+ 66us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.641608 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":111,"threads_started":1}
I20251214 03:02:56.641824 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.637578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41383 (local address 127.0.0.1:50752)
1214 03:02:56.638170 (+ 592us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.638183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.638277 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.639400 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.639404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.639424 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:56.639816 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.639824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.640798 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.640801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.641467 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.641477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.641584 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.641602 (+ 18us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.641648 (+ 46us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.641705 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":104,"threads_started":1}
I20251214 03:02:56.642428 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.637815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.1.157.195:41383)
1214 03:02:56.639020 (+ 1205us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.639025 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.639043 (+ 18us) server_negotiation.cc:408] Connection header received
1214 03:02:56.639111 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.639115 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.639262 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.639403 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:56.639925 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.640681 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.641597 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.642072 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.642165 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.642219 (+ 54us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.642297 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1106,"thread_start_us":90,"threads_started":1}
I20251214 03:02:56.644402 9340 tablet_service.cc:1505] Processing CreateTablet for tablet b743c53920ef47fc8f841d49390f7d8c (DEFAULT_TABLE table=test-table [id=39879710319949f0ba9599e76fdfd717]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:56.644614 9612 tablet_service.cc:1505] Processing CreateTablet for tablet b743c53920ef47fc8f841d49390f7d8c (DEFAULT_TABLE table=test-table [id=39879710319949f0ba9599e76fdfd717]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:56.645320 9340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b743c53920ef47fc8f841d49390f7d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:56.645447 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b743c53920ef47fc8f841d49390f7d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:56.646576 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.637894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:43857 (local address 127.0.0.1:56682)
1214 03:02:56.638693 (+ 799us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.638703 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.638785 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.639708 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.639711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.639726 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:56.639966 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.639975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.645303 (+ 5328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.645308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.646184 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.646194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.646313 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.646332 (+ 19us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.646384 (+ 52us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.646442 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":84,"threads_started":1}
I20251214 03:02:56.647388 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.638041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.1.157.194:43857)
1214 03:02:56.638338 (+ 297us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.638344 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.638783 (+ 439us) server_negotiation.cc:408] Connection header received
1214 03:02:56.638908 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.638913 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.639575 (+ 662us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.639708 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:56.640818 (+ 1110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.643000 (+ 2182us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.646339 (+ 3339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.646999 (+ 660us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.647098 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.647162 (+ 64us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.647254 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20251214 03:02:56.650210 9476 tablet_service.cc:1505] Processing CreateTablet for tablet b743c53920ef47fc8f841d49390f7d8c (DEFAULT_TABLE table=test-table [id=39879710319949f0ba9599e76fdfd717]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:02:56.651225 9476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b743c53920ef47fc8f841d49390f7d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:02:56.651906 9712 tablet_bootstrap.cc:492] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: Bootstrap starting.
I20251214 03:02:56.652933 9713 tablet_bootstrap.cc:492] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: Bootstrap starting.
I20251214 03:02:56.654697 9712 tablet_bootstrap.cc:654] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:56.654963 9713 tablet_bootstrap.cc:654] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:56.655561 9712 log.cc:826] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:56.655645 9713 log.cc:826] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:56.657473 9712 tablet_bootstrap.cc:492] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: No bootstrap required, opened a new log
I20251214 03:02:56.657680 9712 ts_tablet_manager.cc:1403] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20251214 03:02:56.657763 9713 tablet_bootstrap.cc:492] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: No bootstrap required, opened a new log
I20251214 03:02:56.657779 9716 tablet_bootstrap.cc:492] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: Bootstrap starting.
I20251214 03:02:56.657934 9713 ts_tablet_manager.cc:1403] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251214 03:02:56.659945 9716 tablet_bootstrap.cc:654] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: Neither blocks nor log segments found. Creating new log.
I20251214 03:02:56.660645 9716 log.cc:826] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: Log is configured to *not* fsync() on all Append() calls
I20251214 03:02:56.660777 9712 raft_consensus.cc:359] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } }
I20251214 03:02:56.661000 9712 raft_consensus.cc:385] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:56.661064 9712 raft_consensus.cc:740] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d4903d11c04204a94556ce1adb4088, State: Initialized, Role: FOLLOWER
I20251214 03:02:56.661573 9712 consensus_queue.cc:260] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } }
I20251214 03:02:56.661994 9713 raft_consensus.cc:359] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } }
I20251214 03:02:56.662192 9713 raft_consensus.cc:385] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:56.662261 9713 raft_consensus.cc:740] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dd607b9ac1a4c779da1359d96d5b5f9, State: Initialized, Role: FOLLOWER
I20251214 03:02:56.662421 9712 ts_tablet_manager.cc:1434] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251214 03:02:56.662570 9716 tablet_bootstrap.cc:492] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: No bootstrap required, opened a new log
I20251214 03:02:56.662730 9716 ts_tablet_manager.cc:1403] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251214 03:02:56.662681 9713 consensus_queue.cc:260] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } }
I20251214 03:02:56.662916 9407 heartbeater.cc:499] Master 127.1.157.254:42577 was elected leader, sending a full tablet report...
I20251214 03:02:56.663475 9713 ts_tablet_manager.cc:1434] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251214 03:02:56.663779 9679 heartbeater.cc:499] Master 127.1.157.254:42577 was elected leader, sending a full tablet report...
I20251214 03:02:56.668320 9716 raft_consensus.cc:359] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } }
I20251214 03:02:56.668550 9716 raft_consensus.cc:385] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:02:56.668612 9716 raft_consensus.cc:740] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3d55d43111412b82f233739185d287, State: Initialized, Role: FOLLOWER
I20251214 03:02:56.669041 9716 consensus_queue.cc:260] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } }
I20251214 03:02:56.669974 9716 ts_tablet_manager.cc:1434] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251214 03:02:56.670225 9543 heartbeater.cc:499] Master 127.1.157.254:42577 was elected leader, sending a full tablet report...
W20251214 03:02:56.720017 9409 tablet.cc:2378] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251214 03:02:56.846891 9681 tablet.cc:2378] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251214 03:02:56.912806 9545 tablet.cc:2378] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 03:02:56.945598 9719 raft_consensus.cc:493] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:02:56.945807 9719 raft_consensus.cc:515] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } }
I20251214 03:02:56.946892 9719 leader_election.cc:290] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74d4903d11c04204a94556ce1adb4088 (127.1.157.193:36175), af3d55d43111412b82f233739185d287 (127.1.157.194:43857)
I20251214 03:02:56.950222 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.947123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:36175 (local address 127.1.157.195:57863)
1214 03:02:56.947312 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.947329 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.947454 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.947750 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.947753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.947773 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:56.947973 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.947979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.948890 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.948892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.949830 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.949839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.949953 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.949974 (+ 21us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.950023 (+ 49us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.950083 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20251214 03:02:56.950222 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.947398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:43857 (local address 127.1.157.195:49513)
1214 03:02:56.947485 (+ 87us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:56.947505 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1214 03:02:56.947565 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:02:56.947841 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:02:56.947844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:02:56.947861 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:02:56.948101 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.948111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.948982 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.948985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:02:56.949830 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:02:56.949839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.949953 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.949974 (+ 21us) client_negotiation.cc:769] Sending connection context
1214 03:02:56.950023 (+ 49us) client_negotiation.cc:240] Negotiation successful
1214 03:02:56.950078 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20251214 03:02:56.950683 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.947233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:57863 (local address 127.1.157.193:36175)
1214 03:02:56.947359 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.947363 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.947391 (+ 28us) server_negotiation.cc:408] Connection header received
1214 03:02:56.947596 (+ 205us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.947600 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.947654 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.947722 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:56.948082 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.948788 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.949941 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.950430 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.950465 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.950516 (+ 51us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.950570 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251214 03:02:56.950683 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:56.947525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:49513 (local address 127.1.157.194:43857)
1214 03:02:56.947662 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:56.947666 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:56.947678 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:02:56.947718 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:56.947721 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:56.947759 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:56.947820 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:02:56.948222 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.948897 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:56.949939 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:56.950427 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:56.950465 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:56.950516 (+ 51us) server_negotiation.cc:300] Negotiation successful
1214 03:02:56.950570 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251214 03:02:56.951238 9495 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b743c53920ef47fc8f841d49390f7d8c" candidate_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3d55d43111412b82f233739185d287" is_pre_election: true
I20251214 03:02:56.951256 9360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b743c53920ef47fc8f841d49390f7d8c" candidate_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4903d11c04204a94556ce1adb4088" is_pre_election: true
I20251214 03:02:56.951512 9360 raft_consensus.cc:2468] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dd607b9ac1a4c779da1359d96d5b5f9 in term 0.
I20251214 03:02:56.951512 9495 raft_consensus.cc:2468] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dd607b9ac1a4c779da1359d96d5b5f9 in term 0.
I20251214 03:02:56.951961 9565 leader_election.cc:304] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d4903d11c04204a94556ce1adb4088, 7dd607b9ac1a4c779da1359d96d5b5f9; no voters:
I20251214 03:02:56.952173 9719 raft_consensus.cc:2804] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:02:56.952247 9719 raft_consensus.cc:493] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:02:56.952276 9719 raft_consensus.cc:3060] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:56.953233 9719 raft_consensus.cc:515] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } }
I20251214 03:02:56.953572 9719 leader_election.cc:290] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [CANDIDATE]: Term 1 election: Requested vote from peers 74d4903d11c04204a94556ce1adb4088 (127.1.157.193:36175), af3d55d43111412b82f233739185d287 (127.1.157.194:43857)
I20251214 03:02:56.953938 9495 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b743c53920ef47fc8f841d49390f7d8c" candidate_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3d55d43111412b82f233739185d287"
I20251214 03:02:56.953958 9360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b743c53920ef47fc8f841d49390f7d8c" candidate_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4903d11c04204a94556ce1adb4088"
I20251214 03:02:56.954046 9495 raft_consensus.cc:3060] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:56.954057 9360 raft_consensus.cc:3060] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:02:56.954890 9360 raft_consensus.cc:2468] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dd607b9ac1a4c779da1359d96d5b5f9 in term 1.
I20251214 03:02:56.954895 9495 raft_consensus.cc:2468] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dd607b9ac1a4c779da1359d96d5b5f9 in term 1.
I20251214 03:02:56.955155 9565 leader_election.cc:304] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dd607b9ac1a4c779da1359d96d5b5f9, af3d55d43111412b82f233739185d287; no voters:
I20251214 03:02:56.955310 9719 raft_consensus.cc:2804] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:02:56.955523 9719 raft_consensus.cc:697] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 7dd607b9ac1a4c779da1359d96d5b5f9, State: Running, Role: LEADER
I20251214 03:02:56.955962 9719 consensus_queue.cc:237] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } }
I20251214 03:02:56.959374 8415 catalog_manager.cc:5654] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7dd607b9ac1a4c779da1359d96d5b5f9 (127.1.157.195). New cstate: current_term: 1 leader_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20251214 03:02:57.006659 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:57.003803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50762 (local address 127.1.157.195:41383)
1214 03:02:57.003937 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:57.003940 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:57.003958 (+ 18us) server_negotiation.cc:408] Connection header received
1214 03:02:57.004048 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:57.004052 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:57.004108 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:57.004188 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:02:57.004467 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:57.004951 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:57.005652 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:57.005812 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:57.005905 (+ 93us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:02:57.006353 (+ 448us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:02:57.006448 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:57.006472 (+ 24us) server_negotiation.cc:300] Negotiation successful
1214 03:02:57.006519 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251214 03:02:57.011983 9495 raft_consensus.cc:1275] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287 [term 1 FOLLOWER]: Refusing update from remote peer 7dd607b9ac1a4c779da1359d96d5b5f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:57.011983 9360 raft_consensus.cc:1275] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088 [term 1 FOLLOWER]: Refusing update from remote peer 7dd607b9ac1a4c779da1359d96d5b5f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:02:57.012663 9719 consensus_queue.cc:1048] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:57.013013 9721 consensus_queue.cc:1048] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:02:57.020534 9724 mvcc.cc:204] Tried to move back new op lower bound from 7232230920233951232 to 7232230920014036992. Current Snapshot: MvccSnapshot[applied={T|T < 7232230920233951232}]
I20251214 03:02:57.021838 9725 mvcc.cc:204] Tried to move back new op lower bound from 7232230920233951232 to 7232230920014036992. Current Snapshot: MvccSnapshot[applied={T|T < 7232230920233951232}]
I20251214 03:02:57.023770 9726 mvcc.cc:204] Tried to move back new op lower bound from 7232230920233951232 to 7232230920014036992. Current Snapshot: MvccSnapshot[applied={T|T < 7232230920233951232}]
I20251214 03:02:57.029942 8414 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:57.030085 8414 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:02:57.034701 8414 catalog_manager.cc:5941] T 00000000000000000000000000000000 P fc546e7566b6428ca27448a8b619ad2b: Sending DeleteTablet for 3 replicas of tablet b743c53920ef47fc8f841d49390f7d8c
I20251214 03:02:57.035713 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet b743c53920ef47fc8f841d49390f7d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:57 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:50752
I20251214 03:02:57.035817 9476 tablet_service.cc:1552] Processing DeleteTablet for tablet b743c53920ef47fc8f841d49390f7d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:57 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:56682
I20251214 03:02:57.035980 9340 tablet_service.cc:1552] Processing DeleteTablet for tablet b743c53920ef47fc8f841d49390f7d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:02:57 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:45366
I20251214 03:02:57.037564 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 8384
I20251214 03:02:57.037806 9736 tablet_replica.cc:333] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: stopping tablet replica
I20251214 03:02:57.038237 9736 raft_consensus.cc:2243] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:57.038591 9736 raft_consensus.cc:2272] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:57.038993 9734 tablet_replica.cc:333] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: stopping tablet replica
I20251214 03:02:57.039283 9734 raft_consensus.cc:2243] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:02:57.039659 9734 raft_consensus.cc:2272] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:57.040031 9735 tablet_replica.cc:333] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: stopping tablet replica
I20251214 03:02:57.040184 9736 ts_tablet_manager.cc:1916] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:57.040287 9735 raft_consensus.cc:2243] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:02:57.040602 9735 raft_consensus.cc:2272] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:02:57.042023 9734 ts_tablet_manager.cc:1916] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:57.043035 9736 ts_tablet_manager.cc:1929] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:02:57.043138 9736 log.cc:1199] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-0/wal/wals/b743c53920ef47fc8f841d49390f7d8c
I20251214 03:02:57.043473 9736 ts_tablet_manager.cc:1950] T b743c53920ef47fc8f841d49390f7d8c P 74d4903d11c04204a94556ce1adb4088: Deleting consensus metadata
I20251214 03:02:57.044565 9734 ts_tablet_manager.cc:1929] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:02:57.044893 9734 log.cc:1199] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/b743c53920ef47fc8f841d49390f7d8c
I20251214 03:02:57.045226 9734 ts_tablet_manager.cc:1950] T b743c53920ef47fc8f841d49390f7d8c P 7dd607b9ac1a4c779da1359d96d5b5f9: Deleting consensus metadata
W20251214 03:02:57.045481 9295 connection.cc:537] server connection from 127.0.0.1:45366 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251214 03:02:57.046190 9566 connection.cc:441] server connection from 127.0.0.1:50752 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50752 (request call id 1) could send its response
I20251214 03:02:57.047128 9735 ts_tablet_manager.cc:1916] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:02:57.051126 9735 ts_tablet_manager.cc:1929] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:02:57.051280 9735 log.cc:1199] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/b743c53920ef47fc8f841d49390f7d8c
I20251214 03:02:57.051644 9735 ts_tablet_manager.cc:1950] T b743c53920ef47fc8f841d49390f7d8c P af3d55d43111412b82f233739185d287: Deleting consensus metadata
W20251214 03:02:57.053135 9430 connection.cc:441] server connection from 127.0.0.1:56682 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56682 (request call id 1) could send its response
I20251214 03:02:57.060312 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:57.049870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.1.157.253:44661)
1214 03:02:57.049998 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:57.050003 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:57.050388 (+ 385us) server_negotiation.cc:408] Connection header received
1214 03:02:57.050626 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:57.050630 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:57.050688 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:57.050777 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:57.052283 (+ 1506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:57.053089 (+ 806us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:57.054184 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:57.054399 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:57.056224 (+ 1825us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:57.056259 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:57.056263 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:57.056293 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:57.058647 (+ 2354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:57.059087 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:57.059091 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:57.059093 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:57.059142 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:57.059426 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:57.059430 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:57.059432 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:57.059641 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:57.059786 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:57.060017 (+ 231us) server_negotiation.cc:300] Negotiation successful
1214 03:02:57.060142 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251214 03:02:57.061717 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:57.053523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.1.157.252:43123)
1214 03:02:57.053637 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1214 03:02:57.053641 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:02:57.053894 (+ 253us) server_negotiation.cc:408] Connection header received
1214 03:02:57.053997 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:02:57.054001 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:02:57.054056 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:02:57.054139 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:02:57.056077 (+ 1938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:57.056578 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:02:57.057314 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:02:57.057522 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:02:57.058539 (+ 1017us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:02:57.058556 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:02:57.058558 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:02:57.058581 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:02:57.060146 (+ 1565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:57.060572 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:57.060575 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:57.060577 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:57.060623 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:02:57.060849 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:02:57.060852 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:02:57.060853 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:02:57.060982 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:02:57.061090 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1214 03:02:57.061494 (+ 404us) server_negotiation.cc:300] Negotiation successful
1214 03:02:57.061607 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251214 03:02:58.026942 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.026504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.195:41301)
1214 03:02:58.026756 (+ 252us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.026835 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20251214 03:02:58.027350 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.026744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.194:35861)
1214 03:02:58.027184 (+ 440us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.027257 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":65,"threads_started":1}
W20251214 03:02:58.027370 9679 heartbeater.cc:646] Failed to heartbeat to 127.1.157.254:42577 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
W20251214 03:02:58.027631 9543 heartbeater.cc:646] Failed to heartbeat to 127.1.157.254:42577 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
I20251214 03:02:58.030496 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.030096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.193:36071)
1214 03:02:58.030357 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.030420 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":64,"threads_started":1}
W20251214 03:02:58.030786 9407 heartbeater.cc:646] Failed to heartbeat to 127.1.157.254:42577 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
I20251214 03:02:58.535902 9746 raft_consensus.cc:493] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fc546e7566b6428ca27448a8b619ad2b)
I20251214 03:02:58.535975 9747 raft_consensus.cc:493] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fc546e7566b6428ca27448a8b619ad2b)
I20251214 03:02:58.536093 9746 raft_consensus.cc:515] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } }
I20251214 03:02:58.536125 9747 raft_consensus.cc:515] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } }
I20251214 03:02:58.537189 9747 leader_election.cc:290] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577), d5053326dfe84c4a97d163a30e8c4d60 (127.1.157.253:44661)
I20251214 03:02:58.537441 9746 leader_election.cc:290] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577), e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252:43123)
I20251214 03:02:58.537725 9221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5053326dfe84c4a97d163a30e8c4d60" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e930f3dd0c0a498d913e4239a7b82416" is_pre_election: true
I20251214 03:02:58.537724 9037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e930f3dd0c0a498d913e4239a7b82416" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d5053326dfe84c4a97d163a30e8c4d60" is_pre_election: true
I20251214 03:02:58.537907 9037 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e930f3dd0c0a498d913e4239a7b82416 in term 1.
I20251214 03:02:58.537907 9221 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5053326dfe84c4a97d163a30e8c4d60 in term 1.
I20251214 03:02:58.537911 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.537356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56644)
1214 03:02:58.537739 (+ 383us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.537802 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":57,"threads_started":1}
W20251214 03:02:58.538090 9197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
I20251214 03:02:58.538300 9198 leader_election.cc:304] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5053326dfe84c4a97d163a30e8c4d60, e930f3dd0c0a498d913e4239a7b82416; no voters:
I20251214 03:02:58.538515 9747 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 03:02:58.538570 9747 raft_consensus.cc:493] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fc546e7566b6428ca27448a8b619ad2b)
I20251214 03:02:58.538605 9747 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 1 FOLLOWER]: Advancing to term 2
I20251214 03:02:58.538595 9012 leader_election.cc:304] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5053326dfe84c4a97d163a30e8c4d60, e930f3dd0c0a498d913e4239a7b82416; no voters:
I20251214 03:02:58.538748 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.538231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56650)
1214 03:02:58.538511 (+ 280us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.538576 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":64,"threads_started":1}
I20251214 03:02:58.538805 9746 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 03:02:58.538883 9746 raft_consensus.cc:493] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fc546e7566b6428ca27448a8b619ad2b)
W20251214 03:02:58.538928 9013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
I20251214 03:02:58.538934 9746 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 1 FOLLOWER]: Advancing to term 2
I20251214 03:02:58.539486 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.539220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56660)
1214 03:02:58.539361 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.539407 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20251214 03:02:58.539638 9197 leader_election.cc:336] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577): Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
I20251214 03:02:58.539815 9747 raft_consensus.cc:515] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } }
I20251214 03:02:58.539880 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.539598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56662)
1214 03:02:58.539772 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.539817 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20251214 03:02:58.539958 9013 leader_election.cc:336] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577): Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
I20251214 03:02:58.540062 9746 raft_consensus.cc:515] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } }
I20251214 03:02:58.540345 9747 leader_election.cc:290] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [CANDIDATE]: Term 2 election: Requested vote from peers fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577), d5053326dfe84c4a97d163a30e8c4d60 (127.1.157.253:44661)
I20251214 03:02:58.540407 9746 leader_election.cc:290] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 2 election: Requested vote from peers fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577), e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252:43123)
I20251214 03:02:58.540740 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.540460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56670)
1214 03:02:58.540611 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.540659 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20251214 03:02:58.540722 9037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e930f3dd0c0a498d913e4239a7b82416" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d5053326dfe84c4a97d163a30e8c4d60"
I20251214 03:02:58.540755 9221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5053326dfe84c4a97d163a30e8c4d60" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e930f3dd0c0a498d913e4239a7b82416"
I20251214 03:02:58.540886 9037 raft_consensus.cc:2393] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e930f3dd0c0a498d913e4239a7b82416 in current term 2: Already voted for candidate d5053326dfe84c4a97d163a30e8c4d60 in this term.
I20251214 03:02:58.540918 9221 raft_consensus.cc:2393] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d5053326dfe84c4a97d163a30e8c4d60 in current term 2: Already voted for candidate e930f3dd0c0a498d913e4239a7b82416 in this term.
I20251214 03:02:58.540972 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.540726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56672)
1214 03:02:58.540870 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.540915 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20251214 03:02:58.541682 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.541486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56694)
1214 03:02:58.541585 (+ 99us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.541630 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20251214 03:02:58.541785 9197 leader_election.cc:336] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577): Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
I20251214 03:02:58.541849 9197 leader_election.cc:304] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e930f3dd0c0a498d913e4239a7b82416; no voters: d5053326dfe84c4a97d163a30e8c4d60, fc546e7566b6428ca27448a8b619ad2b
I20251214 03:02:58.541983 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:58.541379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56680)
1214 03:02:58.541860 (+ 481us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:58.541907 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":53,"threads_started":1}
I20251214 03:02:58.541980 9747 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251214 03:02:58.542090 9013 leader_election.cc:336] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577): Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
I20251214 03:02:58.542151 9013 leader_election.cc:304] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5053326dfe84c4a97d163a30e8c4d60; no voters: e930f3dd0c0a498d913e4239a7b82416, fc546e7566b6428ca27448a8b619ad2b
I20251214 03:02:58.542268 9746 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 03:02:59.028460 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:59.027929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.195:60161)
1214 03:02:59.028285 (+ 356us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:59.028352 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":95,"threads_started":1}
I20251214 03:02:59.028604 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:59.028185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.194:37359)
1214 03:02:59.028451 (+ 266us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:59.028506 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":70,"threads_started":1}
I20251214 03:02:59.031785 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:02:59.031353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.193:35179)
1214 03:02:59.031611 (+ 258us) negotiation.cc:107] Waiting for socket to connect
1214 03:02:59.031669 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
I20251214 03:03:00.029778 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.029266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.195:54685)
1214 03:03:00.029605 (+ 339us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.029669 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":81,"threads_started":1}
I20251214 03:03:00.029872 9757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.029434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.194:42503)
1214 03:03:00.029714 (+ 280us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.029772 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20251214 03:03:00.032796 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.032430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.193:41117)
1214 03:03:00.032648 (+ 218us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.032704 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20251214 03:03:00.105363 9759 raft_consensus.cc:493] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:03:00.105515 9759 raft_consensus.cc:515] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } }
I20251214 03:03:00.105943 9759 leader_election.cc:290] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577), e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252:43123)
I20251214 03:03:00.106328 9221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5053326dfe84c4a97d163a30e8c4d60" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e930f3dd0c0a498d913e4239a7b82416" is_pre_election: true
I20251214 03:03:00.106467 9221 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5053326dfe84c4a97d163a30e8c4d60 in term 2.
I20251214 03:03:00.106499 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.106121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56954)
1214 03:03:00.106347 (+ 226us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.106402 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":56,"threads_started":1}
I20251214 03:03:00.106770 9012 leader_election.cc:304] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5053326dfe84c4a97d163a30e8c4d60, e930f3dd0c0a498d913e4239a7b82416; no voters:
I20251214 03:03:00.106941 9759 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 03:03:00.107007 9759 raft_consensus.cc:493] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:03:00.107045 9759 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 2 FOLLOWER]: Advancing to term 3
I20251214 03:03:00.107313 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.107065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56968)
1214 03:03:00.107185 (+ 120us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.107244 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20251214 03:03:00.107445 9013 leader_election.cc:336] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577): Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
I20251214 03:03:00.108080 9759 raft_consensus.cc:515] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } }
I20251214 03:03:00.108392 9759 leader_election.cc:290] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 3 election: Requested vote from peers fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577), e930f3dd0c0a498d913e4239a7b82416 (127.1.157.252:43123)
I20251214 03:03:00.108816 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.108537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56970)
1214 03:03:00.108711 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.108759 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20251214 03:03:00.108855 9221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5053326dfe84c4a97d163a30e8c4d60" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e930f3dd0c0a498d913e4239a7b82416"
I20251214 03:03:00.109241 9221 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 2 FOLLOWER]: Advancing to term 3
I20251214 03:03:00.110330 9221 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5053326dfe84c4a97d163a30e8c4d60 in term 3.
I20251214 03:03:00.110602 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.110117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56978)
1214 03:03:00.110276 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.110526 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20251214 03:03:00.110880 9012 leader_election.cc:304] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5053326dfe84c4a97d163a30e8c4d60, e930f3dd0c0a498d913e4239a7b82416; no voters:
W20251214 03:03:00.111073 9013 leader_election.cc:336] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577): Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
I20251214 03:03:00.111286 9759 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 03:03:00.111466 9759 raft_consensus.cc:697] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [term 3 LEADER]: Becoming Leader. State: Replica: d5053326dfe84c4a97d163a30e8c4d60, State: Running, Role: LEADER
I20251214 03:03:00.111724 9759 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } }
I20251214 03:03:00.112720 9762 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5053326dfe84c4a97d163a30e8c4d60. Latest consensus state: current_term: 3 leader_uuid: "d5053326dfe84c4a97d163a30e8c4d60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:03:00.112836 9762 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:00.113108 9764 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:03:00.115173 9764 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:03:00.115558 9764 catalog_manager.cc:1269] Loaded cluster ID: 4fcd4b0673a74ba5a0393d9fa9c2a01e
I20251214 03:03:00.115602 9764 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:03:00.116117 9764 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:03:00.116490 9764 catalog_manager.cc:6038] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Loaded TSK: 0
I20251214 03:03:00.117106 9764 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 03:03:00.180320 9026 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:03:00.182636 9026 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:03:00.185297 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.185044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:56998)
1214 03:03:00.185146 (+ 102us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.185221 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20251214 03:03:00.185375 9221 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [term 3 FOLLOWER]: Refusing update from remote peer d5053326dfe84c4a97d163a30e8c4d60: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251214 03:03:00.185783 9762 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251214 03:03:00.185926 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.185724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.0.0.1:57006)
1214 03:03:00.185824 (+ 100us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.185866 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20251214 03:03:00.186074 9013 consensus_peers.cc:597] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 -> Peer fc546e7566b6428ca27448a8b619ad2b (127.1.157.254:42577): Couldn't send request to peer fc546e7566b6428ca27448a8b619ad2b. Status: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251214 03:03:00.187126 9766 sys_catalog.cc:455] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5053326dfe84c4a97d163a30e8c4d60. Latest consensus state: current_term: 3 leader_uuid: "d5053326dfe84c4a97d163a30e8c4d60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:03:00.187270 9766 sys_catalog.cc:458] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 03:03:00.188283 9762 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d5053326dfe84c4a97d163a30e8c4d60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:03:00.188385 9762 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60 [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:00.188552 9766 sys_catalog.cc:455] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d5053326dfe84c4a97d163a30e8c4d60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fc546e7566b6428ca27448a8b619ad2b" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42577 } } peers { permanent_uuid: "d5053326dfe84c4a97d163a30e8c4d60" member_type: VOTER last_known_addr { host: "127.1.157.253" port: 44661 } attrs { promote: false } } peers { permanent_uuid: "e930f3dd0c0a498d913e4239a7b82416" member_type: VOTER last_known_addr { host: "127.1.157.252" port: 43123 } attrs { promote: false } } }
I20251214 03:03:00.188711 9766 sys_catalog.cc:458] T 00000000000000000000000000000000 P e930f3dd0c0a498d913e4239a7b82416 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 03:03:00.197609 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.194092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:43857 (local address 127.0.0.1:56686)
1214 03:03:00.194200 (+ 108us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.194216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:00.194301 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:00.195402 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:00.195405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:00.195424 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:00.195671 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:00.195677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.196521 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.196524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:00.197215 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:00.197223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.197322 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.197342 (+ 20us) client_negotiation.cc:769] Sending connection context
1214 03:03:00.197406 (+ 64us) client_negotiation.cc:240] Negotiation successful
1214 03:03:00.197483 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20251214 03:03:00.198073 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.194246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.1.157.194:43857)
1214 03:03:00.195116 (+ 870us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:00.195120 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:00.195135 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:03:00.195185 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:00.195188 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:00.195250 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:00.195377 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:00.195795 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.196412 (+ 617us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.197320 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.197802 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.197840 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:00.197890 (+ 50us) server_negotiation.cc:300] Negotiation successful
1214 03:03:00.197950 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":61,"threads_started":1}
I20251214 03:03:00.198171 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.194690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:36175 (local address 127.0.0.1:45378)
1214 03:03:00.195367 (+ 677us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.195378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:00.195451 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:00.195743 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:00.195746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:00.195762 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:00.195987 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:00.195993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.196997 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.197001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:00.197833 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:00.197844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.197947 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.197962 (+ 15us) client_negotiation.cc:769] Sending connection context
1214 03:03:00.198008 (+ 46us) client_negotiation.cc:240] Negotiation successful
1214 03:03:00.198063 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":117,"threads_started":1}
I20251214 03:03:00.198458 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.194733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41383 (local address 127.0.0.1:50770)
1214 03:03:00.195439 (+ 706us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.195460 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:00.195542 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:00.195840 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:00.195842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:00.195856 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:00.196026 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:00.196030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.197254 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.197258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:00.198117 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:00.198127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.198231 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.198247 (+ 16us) client_negotiation.cc:769] Sending connection context
1214 03:03:00.198292 (+ 45us) client_negotiation.cc:240] Negotiation successful
1214 03:03:00.198347 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":85,"threads_started":1}
I20251214 03:03:00.198647 9476 tablet_service.cc:1505] Processing CreateTablet for tablet 4bdcfbc2cfb14171a32ec308d066c27c (DEFAULT_TABLE table=test-table [id=3e4e2f4b33f44ec7966df425864908af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:00.198935 9476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bdcfbc2cfb14171a32ec308d066c27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:00.199190 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.194841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.1.157.195:41383)
1214 03:03:00.195035 (+ 194us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:00.195038 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:00.195527 (+ 489us) server_negotiation.cc:408] Connection header received
1214 03:03:00.195621 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:00.195625 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:00.195676 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:00.195748 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:00.196147 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.196894 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.198448 (+ 1554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.198921 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.198956 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:00.199007 (+ 51us) server_negotiation.cc:300] Negotiation successful
1214 03:03:00.199068 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20251214 03:03:00.199410 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.194810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.1.157.193:36175)
1214 03:03:00.195267 (+ 457us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:00.195271 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:00.195535 (+ 264us) server_negotiation.cc:408] Connection header received
1214 03:03:00.195591 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:00.195594 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:00.195642 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:00.195718 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:00.196097 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.196885 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.198682 (+ 1797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.199138 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.199170 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:00.199217 (+ 47us) server_negotiation.cc:300] Negotiation successful
1214 03:03:00.199284 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":59,"threads_started":1}
I20251214 03:03:00.199872 9340 tablet_service.cc:1505] Processing CreateTablet for tablet 4bdcfbc2cfb14171a32ec308d066c27c (DEFAULT_TABLE table=test-table [id=3e4e2f4b33f44ec7966df425864908af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:00.199983 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 4bdcfbc2cfb14171a32ec308d066c27c (DEFAULT_TABLE table=test-table [id=3e4e2f4b33f44ec7966df425864908af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:00.200121 9340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bdcfbc2cfb14171a32ec308d066c27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:00.200196 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bdcfbc2cfb14171a32ec308d066c27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:00.201189 9776 tablet_bootstrap.cc:492] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287: Bootstrap starting.
I20251214 03:03:00.202150 9776 tablet_bootstrap.cc:654] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:00.202263 9777 tablet_bootstrap.cc:492] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088: Bootstrap starting.
I20251214 03:03:00.202946 9778 tablet_bootstrap.cc:492] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9: Bootstrap starting.
I20251214 03:03:00.203254 9777 tablet_bootstrap.cc:654] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:00.203362 9776 tablet_bootstrap.cc:492] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287: No bootstrap required, opened a new log
I20251214 03:03:00.203457 9776 ts_tablet_manager.cc:1403] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:00.203873 9778 tablet_bootstrap.cc:654] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:00.203893 9776 raft_consensus.cc:359] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } }
I20251214 03:03:00.204001 9776 raft_consensus.cc:385] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:00.204041 9776 raft_consensus.cc:740] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3d55d43111412b82f233739185d287, State: Initialized, Role: FOLLOWER
I20251214 03:03:00.204165 9776 consensus_queue.cc:260] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } }
I20251214 03:03:00.204355 9777 tablet_bootstrap.cc:492] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088: No bootstrap required, opened a new log
I20251214 03:03:00.204430 9777 ts_tablet_manager.cc:1403] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251214 03:03:00.204493 9776 ts_tablet_manager.cc:1434] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:03:00.204811 9777 raft_consensus.cc:359] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } }
I20251214 03:03:00.204968 9777 raft_consensus.cc:385] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:00.205042 9777 raft_consensus.cc:740] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d4903d11c04204a94556ce1adb4088, State: Initialized, Role: FOLLOWER
I20251214 03:03:00.205073 9757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.204792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.194:56913)
1214 03:03:00.204954 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.205005 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20251214 03:03:00.205252 9777 consensus_queue.cc:260] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } }
I20251214 03:03:00.205610 9777 ts_tablet_manager.cc:1434] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251214 03:03:00.205730 9757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.205496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.194:42639)
1214 03:03:00.205631 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.205676 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20251214 03:03:00.206337 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.206143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.193:53081)
1214 03:03:00.206226 (+ 83us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.206273 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20251214 03:03:00.208014 9408 heartbeater.cc:499] Master 127.1.157.253:44661 was elected leader, sending a full tablet report...
I20251214 03:03:00.208293 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.208058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.193:48989)
1214 03:03:00.208191 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.208229 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20251214 03:03:00.208859 9544 heartbeater.cc:499] Master 127.1.157.253:44661 was elected leader, sending a full tablet report...
I20251214 03:03:00.209803 9778 tablet_bootstrap.cc:492] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9: No bootstrap required, opened a new log
I20251214 03:03:00.210031 9778 ts_tablet_manager.cc:1403] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251214 03:03:00.210446 9778 raft_consensus.cc:359] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } }
I20251214 03:03:00.210537 9778 raft_consensus.cc:385] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:00.210567 9778 raft_consensus.cc:740] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dd607b9ac1a4c779da1359d96d5b5f9, State: Initialized, Role: FOLLOWER
I20251214 03:03:00.210698 9778 consensus_queue.cc:260] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } }
I20251214 03:03:00.211030 9778 ts_tablet_manager.cc:1434] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:03:00.211557 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.211347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.195:43383)
1214 03:03:00.211458 (+ 111us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.211500 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251214 03:03:00.212188 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.211958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.195:44503)
1214 03:03:00.212091 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.212133 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20251214 03:03:00.357434 9781 raft_consensus.cc:493] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:03:00.357589 9781 raft_consensus.cc:515] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } }
I20251214 03:03:00.358659 9781 leader_election.cc:290] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7dd607b9ac1a4c779da1359d96d5b5f9 (127.1.157.195:41383), 74d4903d11c04204a94556ce1adb4088 (127.1.157.193:36175)
I20251214 03:03:00.361903 9757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.358916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:36175 (local address 127.1.157.194:36625)
1214 03:03:00.359145 (+ 229us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.359189 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:00.359291 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:00.359563 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:00.359566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:00.359582 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:00.359839 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:00.359846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.360882 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.360886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:00.361488 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:00.361496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.361583 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.361605 (+ 22us) client_negotiation.cc:769] Sending connection context
1214 03:03:00.361652 (+ 47us) client_negotiation.cc:240] Negotiation successful
1214 03:03:00.361715 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20251214 03:03:00.361954 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.358916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41383 (local address 127.1.157.194:33749)
1214 03:03:00.359295 (+ 379us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.359307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:00.359417 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:00.359655 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:00.359658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:00.359673 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:00.360207 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:00.360216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.360911 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.360914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:00.361522 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:00.361530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.361650 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.361668 (+ 18us) client_negotiation.cc:769] Sending connection context
1214 03:03:00.361718 (+ 50us) client_negotiation.cc:240] Negotiation successful
1214 03:03:00.361770 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":37,"thread_start_us":117,"threads_started":1}
I20251214 03:03:00.362311 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.359001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:36625 (local address 127.1.157.193:36175)
1214 03:03:00.359134 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:00.359138 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:00.359251 (+ 113us) server_negotiation.cc:408] Connection header received
1214 03:03:00.359393 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:00.359396 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:00.359448 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:00.359531 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:00.359959 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.360770 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.361608 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.362053 (+ 445us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.362093 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:00.362143 (+ 50us) server_negotiation.cc:300] Negotiation successful
1214 03:03:00.362202 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251214 03:03:00.362387 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.359010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:33749 (local address 127.1.157.195:41383)
1214 03:03:00.359145 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:00.359149 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:00.359399 (+ 250us) server_negotiation.cc:408] Connection header received
1214 03:03:00.359498 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:00.359508 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:00.359563 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:00.359636 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:00.360333 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.360796 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.361648 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.362119 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.362157 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:00.362240 (+ 83us) server_negotiation.cc:300] Negotiation successful
1214 03:03:00.362297 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251214 03:03:00.362726 9360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bdcfbc2cfb14171a32ec308d066c27c" candidate_uuid: "af3d55d43111412b82f233739185d287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4903d11c04204a94556ce1adb4088" is_pre_election: true
I20251214 03:03:00.362805 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bdcfbc2cfb14171a32ec308d066c27c" candidate_uuid: "af3d55d43111412b82f233739185d287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" is_pre_election: true
I20251214 03:03:00.362870 9360 raft_consensus.cc:2468] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3d55d43111412b82f233739185d287 in term 0.
I20251214 03:03:00.363008 9632 raft_consensus.cc:2468] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3d55d43111412b82f233739185d287 in term 0.
I20251214 03:03:00.363247 9429 leader_election.cc:304] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d4903d11c04204a94556ce1adb4088, af3d55d43111412b82f233739185d287; no voters:
I20251214 03:03:00.363456 9781 raft_consensus.cc:2804] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:03:00.363507 9781 raft_consensus.cc:493] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:03:00.363534 9781 raft_consensus.cc:3060] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:00.364375 9781 raft_consensus.cc:515] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } }
I20251214 03:03:00.364739 9781 leader_election.cc:290] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [CANDIDATE]: Term 1 election: Requested vote from peers 7dd607b9ac1a4c779da1359d96d5b5f9 (127.1.157.195:41383), 74d4903d11c04204a94556ce1adb4088 (127.1.157.193:36175)
I20251214 03:03:00.365069 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bdcfbc2cfb14171a32ec308d066c27c" candidate_uuid: "af3d55d43111412b82f233739185d287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9"
I20251214 03:03:00.365095 9360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bdcfbc2cfb14171a32ec308d066c27c" candidate_uuid: "af3d55d43111412b82f233739185d287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4903d11c04204a94556ce1adb4088"
I20251214 03:03:00.365200 9632 raft_consensus.cc:3060] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:00.365221 9360 raft_consensus.cc:3060] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:00.365962 9360 raft_consensus.cc:2468] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3d55d43111412b82f233739185d287 in term 1.
I20251214 03:03:00.366110 9632 raft_consensus.cc:2468] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3d55d43111412b82f233739185d287 in term 1.
I20251214 03:03:00.366279 9429 leader_election.cc:304] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d4903d11c04204a94556ce1adb4088, af3d55d43111412b82f233739185d287; no voters:
I20251214 03:03:00.366472 9781 raft_consensus.cc:2804] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:00.366712 9781 raft_consensus.cc:697] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 1 LEADER]: Becoming Leader. State: Replica: af3d55d43111412b82f233739185d287, State: Running, Role: LEADER
I20251214 03:03:00.367102 9781 consensus_queue.cc:237] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } } peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } }
I20251214 03:03:00.367370 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.367147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.194:57901)
1214 03:03:00.367259 (+ 112us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.367314 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20251214 03:03:00.369621 9025 catalog_manager.cc:5654] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3d55d43111412b82f233739185d287 (127.1.157.194). New cstate: current_term: 1 leader_uuid: "af3d55d43111412b82f233739185d287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3d55d43111412b82f233739185d287" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 } health_report { overall_health: UNKNOWN } } }
I20251214 03:03:00.390230 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.387029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.1.157.194:43857)
1214 03:03:00.387157 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:00.387161 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:00.387176 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:03:00.387256 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:00.387259 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:00.387314 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:00.387410 (+ 96us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:00.387736 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.388295 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:00.389039 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:00.389296 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:00.389362 (+ 66us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:00.389842 (+ 480us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:00.389947 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:00.390050 (+ 103us) server_negotiation.cc:300] Negotiation successful
1214 03:03:00.390107 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251214 03:03:00.393206 9360 raft_consensus.cc:1275] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088 [term 1 FOLLOWER]: Refusing update from remote peer af3d55d43111412b82f233739185d287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:00.393429 9632 raft_consensus.cc:1275] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 1 FOLLOWER]: Refusing update from remote peer af3d55d43111412b82f233739185d287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:00.393853 9781 consensus_queue.cc:1048] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d4903d11c04204a94556ce1adb4088" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 36175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:00.394037 9786 consensus_queue.cc:1048] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dd607b9ac1a4c779da1359d96d5b5f9" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:00.395267 9680 heartbeater.cc:499] Master 127.1.157.253:44661 was elected leader, sending a full tablet report...
I20251214 03:03:00.395519 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.395192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.193:60103)
1214 03:03:00.395380 (+ 188us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.395438 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20251214 03:03:00.395792 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.395548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.195:40149)
1214 03:03:00.395666 (+ 118us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.395725 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20251214 03:03:00.398757 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.398201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.194:46499)
1214 03:03:00.398632 (+ 431us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.398686 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358}
I20251214 03:03:00.399374 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.399094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.193:53109)
1214 03:03:00.399259 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.399308 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20251214 03:03:00.400456 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.399094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.194:51427)
1214 03:03:00.400356 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.400398 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1196}
I20251214 03:03:00.401057 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:00.400750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42577 (local address 127.1.157.195:49511)
1214 03:03:00.400963 (+ 213us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.401002 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20251214 03:03:00.406586 9027 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41980:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:00.406737 9027 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41980:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:00.410107 9027 catalog_manager.cc:5941] T 00000000000000000000000000000000 P d5053326dfe84c4a97d163a30e8c4d60: Sending DeleteTablet for 3 replicas of tablet 4bdcfbc2cfb14171a32ec308d066c27c
I20251214 03:03:00.410615 9476 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bdcfbc2cfb14171a32ec308d066c27c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:00 UTC) from {username='oryx', principal='oryx/127.1.157.253@KRBTEST.COM'} at 127.0.0.1:56686
I20251214 03:03:00.410712 9340 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bdcfbc2cfb14171a32ec308d066c27c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:00 UTC) from {username='oryx', principal='oryx/127.1.157.253@KRBTEST.COM'} at 127.0.0.1:45378
I20251214 03:03:00.410899 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bdcfbc2cfb14171a32ec308d066c27c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:00 UTC) from {username='oryx', principal='oryx/127.1.157.253@KRBTEST.COM'} at 127.0.0.1:50770
I20251214 03:03:00.411142 9801 tablet_replica.cc:333] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088: stopping tablet replica
I20251214 03:03:00.411223 9802 tablet_replica.cc:333] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9: stopping tablet replica
I20251214 03:03:00.411255 9801 raft_consensus.cc:2243] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:00.411326 9802 raft_consensus.cc:2243] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:00.411375 9801 raft_consensus.cc:2272] T 4bdcfbc2cfb14171a32ec308d066c27c P 74d4903d11c04204a94556ce1adb4088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:00.411439 9802 raft_consensus.cc:2272] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:00.416110 9800 tablet_replica.cc:333] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287: stopping tablet replica
I20251214 03:03:00.416354 9800 raft_consensus.cc:2243] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:03:00.416893 9800 raft_consensus.cc:2272] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:00.417163 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 9277
I20251214 03:03:00.417690 9802 ts_tablet_manager.cc:1916] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:00.418931 9800 ts_tablet_manager.cc:1916] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:00.419948 9802 ts_tablet_manager.cc:1929] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:03:00.420004 9802 log.cc:1199] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/4bdcfbc2cfb14171a32ec308d066c27c
I20251214 03:03:00.420248 9802 ts_tablet_manager.cc:1950] T 4bdcfbc2cfb14171a32ec308d066c27c P 7dd607b9ac1a4c779da1359d96d5b5f9: Deleting consensus metadata
I20251214 03:03:00.420832 9013 catalog_manager.cc:4985] TS 7dd607b9ac1a4c779da1359d96d5b5f9 (127.1.157.195:41383): tablet 4bdcfbc2cfb14171a32ec308d066c27c (table test-table [id=3e4e2f4b33f44ec7966df425864908af]) successfully deleted
I20251214 03:03:00.421093 9800 ts_tablet_manager.cc:1929] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:03:00.421154 9800 log.cc:1199] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/4bdcfbc2cfb14171a32ec308d066c27c
I20251214 03:03:00.421435 9800 ts_tablet_manager.cc:1950] T 4bdcfbc2cfb14171a32ec308d066c27c P af3d55d43111412b82f233739185d287: Deleting consensus metadata
I20251214 03:03:00.422031 9011 catalog_manager.cc:4985] TS af3d55d43111412b82f233739185d287 (127.1.157.194:43857): tablet 4bdcfbc2cfb14171a32ec308d066c27c (table test-table [id=3e4e2f4b33f44ec7966df425864908af]) successfully deleted
W20251214 03:03:00.424932 9772 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:00.424574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:36175 (local address 127.0.0.1:45390)
1214 03:03:00.424683 (+ 109us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:00.424697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:00.424786 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:00.424861 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.193:36175: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":44}
I20251214 03:03:00.425045 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 9413
W20251214 03:03:00.425072 9011 catalog_manager.cc:4712] TS 74d4903d11c04204a94556ce1adb4088 (127.1.157.193:36175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bdcfbc2cfb14171a32ec308d066c27c: Network error: Client connection negotiation failed: client connection to 127.1.157.193:36175: BlockingWrite error: write error: Connection reset by peer (error 104)
I20251214 03:03:00.429836 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 9549
I20251214 03:03:00.435370 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 8888
I20251214 03:03:00.440389 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 9005
2025-12-14T03:03:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10027 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:03:00 dist-test-slave-k7z7 krb5kdc[9805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:03:00 dist-test-slave-k7z7 krb5kdc[9805](info): set up 2 sockets
Dec 14 03:03:00 dist-test-slave-k7z7 krb5kdc[9805](info): commencing operation
krb5kdc: starting...
W20251214 03:03:02.277637 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.816s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:03:02 dist-test-slave-k7z7 krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:03:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:03:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:03:02.408582 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:36667
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:02.510988 9821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:02.511231 9821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:02.511291 9821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:02.514550 9821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:03:02.514605 9821 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251214 03:03:02.514622 9821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:02.514657 9821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:02.514689 9821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:03:02.514717 9821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:03:02.518885 9821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:36667
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:02.519888 9821 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:02.520732 9821 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:02.526113 9827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:02.526190 9826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:02.526187 9821 server_base.cc:1047] running on GCE node
W20251214 03:03:02.526394 9829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:02.526835 9821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:02.527693 9821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:02.528844 9821 hybrid_clock.cc:648] HybridClock initialized: now 1765681382528824 us; error 38 us; skew 500 ppm
Dec 14 03:03:02 dist-test-slave-k7z7 krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681382, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:02.531301 9821 init.cc:377] Logged in from keytab as oryx/127.1.157.254@KRBTEST.COM (short username oryx)
I20251214 03:03:02.532351 9821 webserver.cc:492] Webserver started at http://127.1.157.254:41217/ using document root <none> and password file <none>
I20251214 03:03:02.532919 9821 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:02.532970 9821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:02.533232 9821 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:02.534888 9821 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "9fef168f76e24c65bb15a9a738c1f3f3"
format_stamp: "Formatted at 2025-12-14 03:03:02 on dist-test-slave-k7z7"
server_key: "aa19b929777693683c7c0952966b4064"
server_key_iv: "0d5445eacd68a4c5ba0e22f264547948"
server_key_version: "encryptionkey@0"
I20251214 03:03:02.535364 9821 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "9fef168f76e24c65bb15a9a738c1f3f3"
format_stamp: "Formatted at 2025-12-14 03:03:02 on dist-test-slave-k7z7"
server_key: "aa19b929777693683c7c0952966b4064"
server_key_iv: "0d5445eacd68a4c5ba0e22f264547948"
server_key_version: "encryptionkey@0"
I20251214 03:03:02.538623 9821 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251214 03:03:02.540546 9836 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:02.541496 9821 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:02.541589 9821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "9fef168f76e24c65bb15a9a738c1f3f3"
format_stamp: "Formatted at 2025-12-14 03:03:02 on dist-test-slave-k7z7"
server_key: "aa19b929777693683c7c0952966b4064"
server_key_iv: "0d5445eacd68a4c5ba0e22f264547948"
server_key_version: "encryptionkey@0"
I20251214 03:03:02.541671 9821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:02.554733 9821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:02.557264 9821 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:02.557415 9821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:02.564803 9821 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:36667
I20251214 03:03:02.564999 9898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:36667 every 8 connection(s)
I20251214 03:03:02.565811 9821 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:03:02.568477 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:02.573700 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 9821
I20251214 03:03:02.573833 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:03:02.574049 1655 external_mini_cluster.cc:1468] Setting key 803393035d5cb94216562378bc416a4e
I20251214 03:03:02.576007 9899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3: Bootstrap starting.
I20251214 03:03:02.578433 9899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:02.579198 9899 log.cc:826] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:02.581163 9899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3: No bootstrap required, opened a new log
Dec 14 03:03:02 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:03:02.584960 9899 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fef168f76e24c65bb15a9a738c1f3f3" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 36667 } }
I20251214 03:03:02.585227 9899 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:02.585328 9899 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fef168f76e24c65bb15a9a738c1f3f3, State: Initialized, Role: FOLLOWER
I20251214 03:03:02.585774 9899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fef168f76e24c65bb15a9a738c1f3f3" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 36667 } }
I20251214 03:03:02.585932 9899 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:03:02.585996 9899 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:03:02.586088 9899 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:02.586653 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:02.575381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.1.157.254:36667)
1214 03:03:02.575798 (+ 417us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:02.575807 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:02.575841 (+ 34us) server_negotiation.cc:408] Connection header received
1214 03:03:02.576417 (+ 576us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:02.576436 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:02.576730 (+ 294us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:02.576995 (+ 265us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:02.578011 (+ 1016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:02.578804 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:02.579630 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:02.579894 (+ 264us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:02.581912 (+ 2018us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:02.581933 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:02.581946 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:02.581985 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:02.583931 (+ 1946us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:02.584334 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:02.584340 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:02.584346 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:02.584430 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:02.584626 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:02.584629 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:02.584630 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:02.585027 (+ 397us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:02.585255 (+ 228us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:02.585815 (+ 560us) server_negotiation.cc:300] Negotiation successful
1214 03:03:02.586034 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
I20251214 03:03:02.586910 9899 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fef168f76e24c65bb15a9a738c1f3f3" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 36667 } }
I20251214 03:03:02.587236 9899 leader_election.cc:304] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fef168f76e24c65bb15a9a738c1f3f3; no voters:
I20251214 03:03:02.587495 9899 leader_election.cc:290] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:03:02.587601 9904 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:02.587860 9904 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [term 1 LEADER]: Becoming Leader. State: Replica: 9fef168f76e24c65bb15a9a738c1f3f3, State: Running, Role: LEADER
I20251214 03:03:02.588186 9904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fef168f76e24c65bb15a9a738c1f3f3" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 36667 } }
I20251214 03:03:02.588693 9899 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:03:02.589672 9906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fef168f76e24c65bb15a9a738c1f3f3. Latest consensus state: current_term: 1 leader_uuid: "9fef168f76e24c65bb15a9a738c1f3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fef168f76e24c65bb15a9a738c1f3f3" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 36667 } } }
I20251214 03:03:02.589792 9906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:02.590076 9905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fef168f76e24c65bb15a9a738c1f3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fef168f76e24c65bb15a9a738c1f3f3" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 36667 } } }
I20251214 03:03:02.590124 9913 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:03:02.590147 9905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3 [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:02.592933 9913 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:03:02.598242 9913 catalog_manager.cc:1357] Generated new cluster ID: c1a5b27f3ffa40f2a2f6076d2d2c8621
I20251214 03:03:02.598316 9913 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:03:02.622506 9913 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:03:02.623829 9913 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:03:02.638155 9913 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3: Generated new TSK 0
I20251214 03:03:02.638971 9913 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:03:02.684289 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:36667
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:02.788041 9927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:02.788276 9927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:02.788359 9927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:02.791724 9927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:02.791797 9927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:02.791867 9927 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251214 03:03:02.791929 9927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:03:02.796087 9927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.157.254:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:02.797083 9927 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:02.797900 9927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:02.804131 9933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:02.804179 9935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:02.804131 9932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:02.804433 9927 server_base.cc:1047] running on GCE node
I20251214 03:03:02.804786 9927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:02.805334 9927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:02.806489 9927 hybrid_clock.cc:648] HybridClock initialized: now 1765681382806479 us; error 26 us; skew 500 ppm
Dec 14 03:03:02 dist-test-slave-k7z7 krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681382, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:02.808964 9927 init.cc:377] Logged in from keytab as oryx/127.1.157.193@KRBTEST.COM (short username oryx)
I20251214 03:03:02.810016 9927 webserver.cc:492] Webserver started at http://127.1.157.193:44197/ using document root <none> and password file <none>
I20251214 03:03:02.810534 9927 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:02.810580 9927 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:02.810729 9927 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:02.812394 9927 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "152aa837d18449d891ba835a3cc0420b"
format_stamp: "Formatted at 2025-12-14 03:03:02 on dist-test-slave-k7z7"
server_key: "629b1f2e2328589be92887601e1567d6"
server_key_iv: "17c587111d9dad40bade002920907d6c"
server_key_version: "encryptionkey@0"
I20251214 03:03:02.812796 9927 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "152aa837d18449d891ba835a3cc0420b"
format_stamp: "Formatted at 2025-12-14 03:03:02 on dist-test-slave-k7z7"
server_key: "629b1f2e2328589be92887601e1567d6"
server_key_iv: "17c587111d9dad40bade002920907d6c"
server_key_version: "encryptionkey@0"
I20251214 03:03:02.815903 9927 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251214 03:03:02.817919 9942 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:02.818871 9927 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:02.818961 9927 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "152aa837d18449d891ba835a3cc0420b"
format_stamp: "Formatted at 2025-12-14 03:03:02 on dist-test-slave-k7z7"
server_key: "629b1f2e2328589be92887601e1567d6"
server_key_iv: "17c587111d9dad40bade002920907d6c"
server_key_version: "encryptionkey@0"
I20251214 03:03:02.819049 9927 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:02.843343 9927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:02.846143 9927 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:02.846307 9927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:02.847666 9927 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:02.847713 9927 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:02.847752 9927 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:02.847784 9927 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 14 03:03:02 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681382, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:03:02.857662 9927 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:34591
I20251214 03:03:02.857677 10057 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:34591 every 8 connection(s)
I20251214 03:03:02.858717 9927 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:03:02.859593 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 9927
I20251214 03:03:02.859699 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:03:02.859922 1655 external_mini_cluster.cc:1468] Setting key 48b13504090272b1c302ad4a343f4dfc
I20251214 03:03:02.860090 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:02.848345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:44163 (local address 127.1.157.254:36667)
1214 03:03:02.848488 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:02.848494 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:02.849197 (+ 703us) server_negotiation.cc:408] Connection header received
1214 03:03:02.850121 (+ 924us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:02.850125 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:02.850188 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:02.850292 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:02.852092 (+ 1800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:02.852932 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:02.853811 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:02.854023 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:02.856637 (+ 2614us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:02.856662 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:02.856666 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:02.856701 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:02.858147 (+ 1446us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:02.858647 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:02.858650 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:02.858651 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:02.858688 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:02.858948 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:02.858950 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:02.858951 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:02.859101 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:02.859192 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:02.859792 (+ 600us) server_negotiation.cc:300] Negotiation successful
1214 03:03:02.859931 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251214 03:03:02.860611 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:02.848591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:36667 (local address 127.1.157.193:44163)
1214 03:03:02.849026 (+ 435us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:02.849068 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:02.849912 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:02.850588 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:02.850599 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:02.851165 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:02.851924 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:02.851941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:02.853064 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:02.853069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:02.853692 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:02.853703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:02.853933 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:02.854602 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:02.854638 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:02.856473 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:02.858260 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:02.858265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:02.858275 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:02.858560 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:02.858760 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:02.858763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:02.858765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:02.858865 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:02.859251 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:02.859257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:02.859529 (+ 272us) client_negotiation.cc:769] Sending connection context
1214 03:03:02.859730 (+ 201us) client_negotiation.cc:240] Negotiation successful
1214 03:03:02.859966 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":123,"threads_started":1}
I20251214 03:03:02.861640 10058 heartbeater.cc:344] Connected to a master server at 127.1.157.254:36667
I20251214 03:03:02.861802 10058 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:02.862345 10058 heartbeater.cc:507] Master 127.1.157.254:36667 requested a full tablet report, sending...
I20251214 03:03:02.863869 9852 ts_manager.cc:194] Registered new tserver with Master: 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591)
I20251214 03:03:02.865402 9852 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:44163
WARNING: no policy specified for oryx/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.194@KRBTEST.COM" created.
I20251214 03:03:02.875766 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:02.869412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:36667 (local address 127.1.157.193:53189)
1214 03:03:02.869755 (+ 343us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:02.869768 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:02.869873 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:02.870136 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:02.870140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:02.870397 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:02.870614 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:02.870621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:02.871410 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:02.871413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:02.871832 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:02.871842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:02.871944 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:02.872523 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:02.872537 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:02.872814 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:02.874549 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:02.874553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:02.874554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:02.874715 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:02.874960 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:02.874963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:02.874964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:02.875013 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:02.875371 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:02.875373 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:02.875445 (+ 72us) client_negotiation.cc:769] Sending connection context
1214 03:03:02.875525 (+ 80us) client_negotiation.cc:240] Negotiation successful
1214 03:03:02.875625 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":138,"threads_started":1}
I20251214 03:03:02.875880 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:02.869506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:53189 (local address 127.1.157.254:36667)
1214 03:03:02.869680 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:02.869685 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:02.869837 (+ 152us) server_negotiation.cc:408] Connection header received
1214 03:03:02.869961 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:02.869965 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:02.870010 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:02.870085 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:02.870749 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:02.871299 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:02.871932 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:02.872110 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:02.872939 (+ 829us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:02.872954 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:02.872958 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:02.872984 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:02.874450 (+ 1466us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:02.874804 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:02.874808 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:02.874810 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:02.874863 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:02.875091 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:02.875095 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:02.875097 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:02.875288 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:02.875385 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:02.875601 (+ 216us) server_negotiation.cc:300] Negotiation successful
1214 03:03:02.875723 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:03:02.912856 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:36667
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:03.020257 10069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:03.020476 10069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:03.020556 10069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:03.023854 10069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:03.023917 10069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:03.023985 10069 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251214 03:03:03.024045 10069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:03:03.028193 10069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.157.254:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:03.029270 10069 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:03.030081 10069 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:03.036701 10074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:03.036701 10075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:03.037367 10077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:03.037478 10069 server_base.cc:1047] running on GCE node
I20251214 03:03:03.037880 10069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:03.038455 10069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:03.039606 10069 hybrid_clock.cc:648] HybridClock initialized: now 1765681383039589 us; error 31 us; skew 500 ppm
Dec 14 03:03:03 dist-test-slave-k7z7 krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681383, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:03.042071 10069 init.cc:377] Logged in from keytab as oryx/127.1.157.194@KRBTEST.COM (short username oryx)
I20251214 03:03:03.043043 10069 webserver.cc:492] Webserver started at http://127.1.157.194:40123/ using document root <none> and password file <none>
I20251214 03:03:03.043561 10069 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:03.043610 10069 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:03.043776 10069 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:03.045523 10069 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "bae10355c88b4e40a02e6016c9b2d260"
format_stamp: "Formatted at 2025-12-14 03:03:03 on dist-test-slave-k7z7"
server_key: "4cfacf4508709bd6984486ea20ed5437"
server_key_iv: "7f411712a1eddeca297d41ddf65be790"
server_key_version: "encryptionkey@0"
I20251214 03:03:03.045977 10069 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "bae10355c88b4e40a02e6016c9b2d260"
format_stamp: "Formatted at 2025-12-14 03:03:03 on dist-test-slave-k7z7"
server_key: "4cfacf4508709bd6984486ea20ed5437"
server_key_iv: "7f411712a1eddeca297d41ddf65be790"
server_key_version: "encryptionkey@0"
I20251214 03:03:03.049207 10069 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:03:03.051365 10084 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:03.052376 10069 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:03.052477 10069 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "bae10355c88b4e40a02e6016c9b2d260"
format_stamp: "Formatted at 2025-12-14 03:03:03 on dist-test-slave-k7z7"
server_key: "4cfacf4508709bd6984486ea20ed5437"
server_key_iv: "7f411712a1eddeca297d41ddf65be790"
server_key_version: "encryptionkey@0"
I20251214 03:03:03.052569 10069 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:03.091254 10069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:03.094091 10069 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:03.094264 10069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:03.095618 10069 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:03.095667 10069 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:03.095721 10069 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:03.095744 10069 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 14 03:03:03 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681383, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:03:03.105368 10069 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:39519
I20251214 03:03:03.105394 10199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:39519 every 8 connection(s)
I20251214 03:03:03.106204 10069 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:03:03.107506 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.096379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:41995 (local address 127.1.157.254:36667)
1214 03:03:03.096526 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.096532 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.097250 (+ 718us) server_negotiation.cc:408] Connection header received
1214 03:03:03.098126 (+ 876us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.098130 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.098193 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.098280 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:03.099808 (+ 1528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.100562 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.101464 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.101648 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.103960 (+ 2312us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:03.103980 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:03.103983 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:03.104015 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:03.105653 (+ 1638us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.106097 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.106101 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.106104 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.106154 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.106441 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.106445 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.106447 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.106643 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:03.106746 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.107273 (+ 527us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.107376 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251214 03:03:03.108070 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.096639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:36667 (local address 127.1.157.194:41995)
1214 03:03:03.097067 (+ 428us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.097106 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.097923 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.098434 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.098445 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.098982 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:03.099653 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.099672 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.100690 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.100695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.101333 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.101342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.101555 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.102220 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:03.102257 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:03.103812 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:03.105759 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:03.105765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:03.105776 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:03.106005 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:03.106239 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:03.106242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:03.106245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:03.106348 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:03.106802 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:03.106808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:03.107042 (+ 234us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.107214 (+ 172us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.107422 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":126,"threads_started":1}
I20251214 03:03:03.109014 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 10069
I20251214 03:03:03.109118 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:03:03.109222 10200 heartbeater.cc:344] Connected to a master server at 127.1.157.254:36667
I20251214 03:03:03.109370 10200 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:03.109479 1655 external_mini_cluster.cc:1468] Setting key 66d0e56f225ab1fcb26eacc00ac77e1d
I20251214 03:03:03.109916 10200 heartbeater.cc:507] Master 127.1.157.254:36667 requested a full tablet report, sending...
I20251214 03:03:03.110895 9852 ts_manager.cc:194] Registered new tserver with Master: bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519)
I20251214 03:03:03.111542 9852 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:41995
I20251214 03:03:03.117123 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.111288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:46069 (local address 127.1.157.254:36667)
1214 03:03:03.111400 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.111404 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.111500 (+ 96us) server_negotiation.cc:408] Connection header received
1214 03:03:03.111590 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.111593 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.111636 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.111707 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:03.112261 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.112891 (+ 630us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.113628 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.113811 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.114567 (+ 756us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:03.114582 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:03.114584 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:03.114604 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:03.115867 (+ 1263us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.116205 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.116208 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.116210 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.116249 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.116437 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.116441 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.116443 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.116599 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:03.116682 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.116896 (+ 214us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.116992 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251214 03:03:03.117137 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.111214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:36667 (local address 127.1.157.194:46069)
1214 03:03:03.111432 (+ 218us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.111443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.111521 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.111729 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.111732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.111899 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:03.112149 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.112158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.113004 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.113007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.113518 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.113527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.113626 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.114164 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:03.114176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:03.114451 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:03.115965 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:03.115969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:03.115971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:03.116124 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:03.116322 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:03.116325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:03.116327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:03.116368 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:03.116683 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:03.116686 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:03.116759 (+ 73us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.116841 (+ 82us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.116962 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab.
I20251214 03:03:03.159426 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:36667
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:03.266667 10211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:03.266892 10211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:03.266957 10211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:03.270274 10211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:03.270342 10211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:03.270397 10211 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251214 03:03:03.270460 10211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:03:03.274673 10211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/krb5kdc/oryx.keytab
--principal=oryx/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.157.254:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:03.275688 10211 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:03.276495 10211 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:03.282959 10217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:03.283052 10211 server_base.cc:1047] running on GCE node
W20251214 03:03:03.282959 10216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:03.282951 10219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:03.283551 10211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:03.284083 10211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:03.285245 10211 hybrid_clock.cc:648] HybridClock initialized: now 1765681383285222 us; error 36 us; skew 500 ppm
Dec 14 03:03:03 dist-test-slave-k7z7 krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681383, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:03.287727 10211 init.cc:377] Logged in from keytab as oryx/127.1.157.195@KRBTEST.COM (short username oryx)
I20251214 03:03:03.288736 10211 webserver.cc:492] Webserver started at http://127.1.157.195:34641/ using document root <none> and password file <none>
I20251214 03:03:03.289273 10211 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:03.289322 10211 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:03.289474 10211 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:03.291158 10211 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "a9a9b80de17e4062846ac8b3f874d6b0"
format_stamp: "Formatted at 2025-12-14 03:03:03 on dist-test-slave-k7z7"
server_key: "680168bd9c2129cbc9818ab3576d9592"
server_key_iv: "d34a87ad7a9c5380a6f1f170a48b5389"
server_key_version: "encryptionkey@0"
I20251214 03:03:03.291581 10211 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "a9a9b80de17e4062846ac8b3f874d6b0"
format_stamp: "Formatted at 2025-12-14 03:03:03 on dist-test-slave-k7z7"
server_key: "680168bd9c2129cbc9818ab3576d9592"
server_key_iv: "d34a87ad7a9c5380a6f1f170a48b5389"
server_key_version: "encryptionkey@0"
I20251214 03:03:03.294792 10211 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:03:03.296852 10226 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:03.297848 10211 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:03.297937 10211 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "a9a9b80de17e4062846ac8b3f874d6b0"
format_stamp: "Formatted at 2025-12-14 03:03:03 on dist-test-slave-k7z7"
server_key: "680168bd9c2129cbc9818ab3576d9592"
server_key_iv: "d34a87ad7a9c5380a6f1f170a48b5389"
server_key_version: "encryptionkey@0"
I20251214 03:03:03.298029 10211 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:03.308104 10211 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:03.310806 10211 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:03.310959 10211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:03.312330 10211 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:03.312381 10211 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:03.312430 10211 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:03.312484 10211 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 14 03:03:03 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681383, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:03:03.322544 10211 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:41545
I20251214 03:03:03.322611 10341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:41545 every 8 connection(s)
I20251214 03:03:03.323616 10211 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:03:03.323915 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.312936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:34817 (local address 127.1.157.254:36667)
1214 03:03:03.313091 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.313097 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.313830 (+ 733us) server_negotiation.cc:408] Connection header received
1214 03:03:03.314694 (+ 864us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.314699 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.314760 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.314837 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:03.316103 (+ 1266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.316759 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.317581 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.317772 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.320194 (+ 2422us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:03.320215 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:03.320217 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:03.320245 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:03.321924 (+ 1679us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.322504 (+ 580us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.322508 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.322510 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.322568 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.322851 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.322854 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.322856 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.323045 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:03.323166 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.323639 (+ 473us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.323758 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251214 03:03:03.324463 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.313230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:36667 (local address 127.1.157.195:34817)
1214 03:03:03.313678 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.313719 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.314476 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.315012 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.315023 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.315431 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:03.315952 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.315966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.316875 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.316880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.317387 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.317397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.317608 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.318260 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:03.318294 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:03.320018 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:03.322052 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:03.322057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:03.322066 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:03.322407 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:03.322671 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:03.322674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:03.322676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:03.322764 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:03.323170 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:03.323176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:03.323402 (+ 226us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.323560 (+ 158us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.323782 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":124,"threads_started":1}
I20251214 03:03:03.325018 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 10211
I20251214 03:03:03.325139 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:03:03.325403 1655 external_mini_cluster.cc:1468] Setting key 422b4297b60b03e1e3aba0997d47bfb8
I20251214 03:03:03.325562 10342 heartbeater.cc:344] Connected to a master server at 127.1.157.254:36667
I20251214 03:03:03.325716 10342 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:03.326237 10342 heartbeater.cc:507] Master 127.1.157.254:36667 requested a full tablet report, sending...
I20251214 03:03:03.327181 9852 ts_manager.cc:194] Registered new tserver with Master: a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545)
I20251214 03:03:03.327823 9852 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:34817
I20251214 03:03:03.332891 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.327459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:54185 (local address 127.1.157.254:36667)
1214 03:03:03.327568 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.327571 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.327774 (+ 203us) server_negotiation.cc:408] Connection header received
1214 03:03:03.327869 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.327873 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.327915 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.327969 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:03.328733 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.329229 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.329782 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.329954 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.330603 (+ 649us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:03.330619 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:03.330622 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:03.330644 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:03.331693 (+ 1049us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.332017 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.332020 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.332021 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.332059 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.332252 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.332255 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.332257 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.332381 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:03.332460 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.332678 (+ 218us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.332769 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251214 03:03:03.332906 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.327467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:36667 (local address 127.1.157.195:54185)
1214 03:03:03.327701 (+ 234us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.327712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.327797 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.328289 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.328292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.328426 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:03.328626 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.328632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.329324 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.329327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.329688 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.329695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.329787 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.330230 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:03.330243 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:03.330487 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:03.331787 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:03.331790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:03.331793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:03.331938 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:03.332131 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:03.332134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:03.332136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:03.332175 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:03.332459 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:03.332462 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:03.332541 (+ 79us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.332622 (+ 81us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.332739 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20251214 03:03:03.338660 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 14 03:03:03 dist-test-slave-k7z7 krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 14 03:03:03 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:03 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
W20251214 03:03:03.358320 9902 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:03.353454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.1.157.254:36667)
1214 03:03:03.353580 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.353584 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.353595 (+ 11us) server_negotiation.cc:408] Connection header received
1214 03:03:03.353631 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.353633 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.353673 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.353743 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:03.354444 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.354929 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.355476 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.355599 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.358204 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42752: BlockingRecv error: recv got EOF from 127.0.0.1:42752 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Dec 14 03:03:03 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:03:03.366201 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.359526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.1.157.254:36667)
1214 03:03:03.359639 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.359642 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.359714 (+ 72us) server_negotiation.cc:408] Connection header received
1214 03:03:03.359797 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.359799 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.359836 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.359898 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:03.360627 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.361101 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.361710 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.361843 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.363631 (+ 1788us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:03.363649 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:03.363651 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:03.363674 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:03.364930 (+ 1256us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.365286 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.365289 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.365291 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.365333 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.365530 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.365534 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.365535 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.365666 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:03.365777 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.365963 (+ 186us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.366056 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251214 03:03:03.368402 9852 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:03:03.370764 9852 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:03:03.382437 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.378140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41545 (local address 127.0.0.1:50856)
1214 03:03:03.378853 (+ 713us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.378889 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.379016 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.379735 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.379741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.379766 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:03.380054 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.380062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.381189 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.381193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.382027 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.382036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.382125 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.382175 (+ 50us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.382262 (+ 87us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.382334 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":71,"threads_started":1}
I20251214 03:03:03.382884 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.378306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:39519 (local address 127.0.0.1:38228)
1214 03:03:03.379602 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.379615 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.379709 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.380094 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.380097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.380113 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:03.380354 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.380362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.381670 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.381674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.382554 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.382564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.382659 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.382674 (+ 15us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.382715 (+ 41us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.382767 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"thread_start_us":85,"threads_started":1}
I20251214 03:03:03.383028 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.378769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:34591 (local address 127.0.0.1:55440)
1214 03:03:03.379881 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.379892 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.379969 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.380368 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.380372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.380390 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:03.380621 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.380628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.381911 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.381915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.382730 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.382737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.382828 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.382840 (+ 12us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.382881 (+ 41us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.382930 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":69,"threads_started":1}
I20251214 03:03:03.383625 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.378315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.1.157.195:41545)
1214 03:03:03.379300 (+ 985us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.379306 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.379329 (+ 23us) server_negotiation.cc:408] Connection header received
1214 03:03:03.379410 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.379415 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.379590 (+ 175us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.379725 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:03.380185 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.381079 (+ 894us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.382744 (+ 1665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.383234 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.383342 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.383404 (+ 62us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.383482 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":96,"threads_started":1}
I20251214 03:03:03.383680 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.378912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.1.157.193:34591)
1214 03:03:03.379353 (+ 441us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.379359 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.379968 (+ 609us) server_negotiation.cc:408] Connection header received
1214 03:03:03.380077 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.380082 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.380247 (+ 165us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.380367 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:03.380730 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.381803 (+ 1073us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.382851 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.383363 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.383444 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.383495 (+ 51us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.383571 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":64,"threads_started":1}
I20251214 03:03:03.384048 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.379119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.1.157.194:39519)
1214 03:03:03.379469 (+ 350us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.379475 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.379698 (+ 223us) server_negotiation.cc:408] Connection header received
1214 03:03:03.379812 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.379817 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.379968 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.380097 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:03.380476 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.381529 (+ 1053us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.383312 (+ 1783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.383724 (+ 412us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.383822 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.383879 (+ 57us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.383945 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20251214 03:03:03.386232 10276 tablet_service.cc:1505] Processing CreateTablet for tablet 09d0771515c64e79bc24984a21435587 (DEFAULT_TABLE table=test-table [id=3fa66c522ffa40538544719cf34ca8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:03.386446 9992 tablet_service.cc:1505] Processing CreateTablet for tablet 09d0771515c64e79bc24984a21435587 (DEFAULT_TABLE table=test-table [id=3fa66c522ffa40538544719cf34ca8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:03.386457 10134 tablet_service.cc:1505] Processing CreateTablet for tablet 09d0771515c64e79bc24984a21435587 (DEFAULT_TABLE table=test-table [id=3fa66c522ffa40538544719cf34ca8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:03.387076 10276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d0771515c64e79bc24984a21435587. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:03.387298 10134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d0771515c64e79bc24984a21435587. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:03.387300 9992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d0771515c64e79bc24984a21435587. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:03.392385 10370 tablet_bootstrap.cc:492] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: Bootstrap starting.
I20251214 03:03:03.393610 10371 tablet_bootstrap.cc:492] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: Bootstrap starting.
I20251214 03:03:03.393610 10372 tablet_bootstrap.cc:492] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: Bootstrap starting.
I20251214 03:03:03.395062 10370 tablet_bootstrap.cc:654] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:03.395254 10372 tablet_bootstrap.cc:654] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:03.395570 10371 tablet_bootstrap.cc:654] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:03.396215 10372 log.cc:826] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:03.396265 10371 log.cc:826] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:03.396261 10370 log.cc:826] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:03.397985 10371 tablet_bootstrap.cc:492] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: No bootstrap required, opened a new log
I20251214 03:03:03.398006 10370 tablet_bootstrap.cc:492] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: No bootstrap required, opened a new log
I20251214 03:03:03.398006 10372 tablet_bootstrap.cc:492] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: No bootstrap required, opened a new log
I20251214 03:03:03.398159 10371 ts_tablet_manager.cc:1403] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251214 03:03:03.398164 10372 ts_tablet_manager.cc:1403] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251214 03:03:03.398164 10370 ts_tablet_manager.cc:1403] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251214 03:03:03.400655 10371 raft_consensus.cc:359] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } }
I20251214 03:03:03.400836 10371 raft_consensus.cc:385] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:03.400907 10371 raft_consensus.cc:740] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152aa837d18449d891ba835a3cc0420b, State: Initialized, Role: FOLLOWER
I20251214 03:03:03.401062 10370 raft_consensus.cc:359] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } }
I20251214 03:03:03.401062 10372 raft_consensus.cc:359] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } }
I20251214 03:03:03.401286 10372 raft_consensus.cc:385] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:03.401286 10370 raft_consensus.cc:385] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:03.401343 10370 raft_consensus.cc:740] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a9b80de17e4062846ac8b3f874d6b0, State: Initialized, Role: FOLLOWER
I20251214 03:03:03.401387 10372 raft_consensus.cc:740] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae10355c88b4e40a02e6016c9b2d260, State: Initialized, Role: FOLLOWER
I20251214 03:03:03.401391 10371 consensus_queue.cc:260] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } }
I20251214 03:03:03.401819 10372 consensus_queue.cc:260] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } }
I20251214 03:03:03.402031 10058 heartbeater.cc:499] Master 127.1.157.254:36667 was elected leader, sending a full tablet report...
I20251214 03:03:03.402235 10371 ts_tablet_manager.cc:1434] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251214 03:03:03.402518 10200 heartbeater.cc:499] Master 127.1.157.254:36667 was elected leader, sending a full tablet report...
I20251214 03:03:03.401819 10370 consensus_queue.cc:260] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } }
I20251214 03:03:03.402776 10372 ts_tablet_manager.cc:1434] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251214 03:03:03.403290 10342 heartbeater.cc:499] Master 127.1.157.254:36667 was elected leader, sending a full tablet report...
I20251214 03:03:03.403440 10370 ts_tablet_manager.cc:1434] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20251214 03:03:03.574729 10343 tablet.cc:2378] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251214 03:03:03.607335 10201 tablet.cc:2378] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251214 03:03:03.610927 10059 tablet.cc:2378] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 03:03:03.688306 10376 raft_consensus.cc:493] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:03:03.688477 10376 raft_consensus.cc:515] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } }
I20251214 03:03:03.689576 10376 leader_election.cc:290] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545), bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519)
I20251214 03:03:03.692658 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.689833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:39519 (local address 127.1.157.193:54141)
1214 03:03:03.690113 (+ 280us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.690128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.690230 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.690499 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.690502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.690521 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:03.690814 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.690822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.691637 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.691640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.692223 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.692231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.692345 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.692368 (+ 23us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.692417 (+ 49us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.692470 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":104,"threads_started":1}
I20251214 03:03:03.692696 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.689837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41545 (local address 127.1.157.193:39895)
1214 03:03:03.690252 (+ 415us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.690292 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.690392 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.690652 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.690656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.690676 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:03.690937 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.690946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.691660 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.691663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.692257 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.692265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.692376 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.692396 (+ 20us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.692454 (+ 58us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.692512 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20251214 03:03:03.693084 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.690090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:54141 (local address 127.1.157.194:39519)
1214 03:03:03.690231 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.690236 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.690257 (+ 21us) server_negotiation.cc:408] Connection header received
1214 03:03:03.690336 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.690340 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.690405 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.690485 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:03.690931 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.691503 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.692359 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.692827 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.692868 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.692923 (+ 55us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.692976 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251214 03:03:03.693089 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.689947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:39895 (local address 127.1.157.195:41545)
1214 03:03:03.690168 (+ 221us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.690173 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.690363 (+ 190us) server_negotiation.cc:408] Connection header received
1214 03:03:03.690494 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.690497 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.690545 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.690623 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:03.691060 (+ 437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.691551 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.692378 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.692839 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.692879 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.692929 (+ 50us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.692983 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20251214 03:03:03.693614 10296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09d0771515c64e79bc24984a21435587" candidate_uuid: "152aa837d18449d891ba835a3cc0420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" is_pre_election: true
I20251214 03:03:03.693635 10154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09d0771515c64e79bc24984a21435587" candidate_uuid: "152aa837d18449d891ba835a3cc0420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae10355c88b4e40a02e6016c9b2d260" is_pre_election: true
I20251214 03:03:03.693846 10296 raft_consensus.cc:2468] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152aa837d18449d891ba835a3cc0420b in term 0.
I20251214 03:03:03.693866 10154 raft_consensus.cc:2468] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152aa837d18449d891ba835a3cc0420b in term 0.
I20251214 03:03:03.694265 9943 leader_election.cc:304] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152aa837d18449d891ba835a3cc0420b, bae10355c88b4e40a02e6016c9b2d260; no voters:
I20251214 03:03:03.694478 10376 raft_consensus.cc:2804] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:03:03.694551 10376 raft_consensus.cc:493] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:03:03.694589 10376 raft_consensus.cc:3060] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:03.695538 10376 raft_consensus.cc:515] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } }
I20251214 03:03:03.695899 10376 leader_election.cc:290] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [CANDIDATE]: Term 1 election: Requested vote from peers a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545), bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519)
I20251214 03:03:03.696175 10296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09d0771515c64e79bc24984a21435587" candidate_uuid: "152aa837d18449d891ba835a3cc0420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a9b80de17e4062846ac8b3f874d6b0"
I20251214 03:03:03.696219 10154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09d0771515c64e79bc24984a21435587" candidate_uuid: "152aa837d18449d891ba835a3cc0420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae10355c88b4e40a02e6016c9b2d260"
I20251214 03:03:03.696283 10296 raft_consensus.cc:3060] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:03.696321 10154 raft_consensus.cc:3060] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:03.697154 10296 raft_consensus.cc:2468] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152aa837d18449d891ba835a3cc0420b in term 1.
I20251214 03:03:03.697235 10154 raft_consensus.cc:2468] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152aa837d18449d891ba835a3cc0420b in term 1.
I20251214 03:03:03.697477 9945 leader_election.cc:304] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152aa837d18449d891ba835a3cc0420b, a9a9b80de17e4062846ac8b3f874d6b0; no voters:
I20251214 03:03:03.697639 10376 raft_consensus.cc:2804] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:03.697831 10376 raft_consensus.cc:697] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 1 LEADER]: Becoming Leader. State: Replica: 152aa837d18449d891ba835a3cc0420b, State: Running, Role: LEADER
I20251214 03:03:03.698124 10376 consensus_queue.cc:237] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } }
I20251214 03:03:03.700999 9847 catalog_manager.cc:5654] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b reported cstate change: term changed from 0 to 1, leader changed from <none> to 152aa837d18449d891ba835a3cc0420b (127.1.157.193). New cstate: current_term: 1 leader_uuid: "152aa837d18449d891ba835a3cc0420b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } health_report { overall_health: HEALTHY } } }
I20251214 03:03:03.743933 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.741085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.1.157.193:34591)
1214 03:03:03.741241 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.741246 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.741262 (+ 16us) server_negotiation.cc:408] Connection header received
1214 03:03:03.741371 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.741374 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.741428 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.741520 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:03.741838 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.742327 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.742927 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.743101 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.743153 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:03.743576 (+ 423us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:03.743672 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.743756 (+ 84us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.743807 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251214 03:03:03.748908 10154 raft_consensus.cc:1275] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Refusing update from remote peer 152aa837d18449d891ba835a3cc0420b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:03.748908 10296 raft_consensus.cc:1275] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Refusing update from remote peer 152aa837d18449d891ba835a3cc0420b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:03.749612 10381 consensus_queue.cc:1048] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:03.749792 10376 consensus_queue.cc:1048] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:03.756616 10386 mvcc.cc:204] Tried to move back new op lower bound from 7232230947829370880 to 7232230947629932544. Current Snapshot: MvccSnapshot[applied={T|T < 7232230947829370880}]
I20251214 03:03:03.763767 9847 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42754:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:03.763906 9847 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42754:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:03.766147 9847 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3: Sending DeleteTablet for 3 replicas of tablet 09d0771515c64e79bc24984a21435587
I20251214 03:03:03.766793 10276 tablet_service.cc:1552] Processing DeleteTablet for tablet 09d0771515c64e79bc24984a21435587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:03 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:50856
I20251214 03:03:03.766921 10134 tablet_service.cc:1552] Processing DeleteTablet for tablet 09d0771515c64e79bc24984a21435587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:03 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:38228
I20251214 03:03:03.767230 10394 tablet_replica.cc:333] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: stopping tablet replica
I20251214 03:03:03.767262 9992 tablet_service.cc:1552] Processing DeleteTablet for tablet 09d0771515c64e79bc24984a21435587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:03 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:55440
I20251214 03:03:03.767503 10394 raft_consensus.cc:2243] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:03.767681 10396 tablet_replica.cc:333] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: stopping tablet replica
I20251214 03:03:03.767796 10394 raft_consensus.cc:2272] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:03.767930 10396 raft_consensus.cc:2243] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:03:03.768268 10396 raft_consensus.cc:2272] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:03.768487 9847 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:03:03.768877 9847 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:03:03.769335 10394 ts_tablet_manager.cc:1916] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:03.769459 10396 ts_tablet_manager.cc:1916] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:03.770308 10395 tablet_replica.cc:333] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: stopping tablet replica
I20251214 03:03:03.770586 10395 raft_consensus.cc:2243] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:03.770917 10395 raft_consensus.cc:2272] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:03.771879 10396 ts_tablet_manager.cc:1929] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:03:03.772001 10396 log.cc:1199] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/wal/wals/09d0771515c64e79bc24984a21435587
I20251214 03:03:03.772308 10396 ts_tablet_manager.cc:1950] T 09d0771515c64e79bc24984a21435587 P 152aa837d18449d891ba835a3cc0420b: Deleting consensus metadata
I20251214 03:03:03.772604 10395 ts_tablet_manager.cc:1916] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:03.773418 9837 catalog_manager.cc:4985] TS 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591): tablet 09d0771515c64e79bc24984a21435587 (table test-table [id=3fa66c522ffa40538544719cf34ca8fd]) successfully deleted
I20251214 03:03:03.774399 9992 tablet_service.cc:1505] Processing CreateTablet for tablet 9a89f25021f7427c815f86026e11ffa8 (DEFAULT_TABLE table=test-table [id=dc87aff5877b41d48bb6247ca836215f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:03.774636 9992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a89f25021f7427c815f86026e11ffa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:03.775131 10134 tablet_service.cc:1505] Processing CreateTablet for tablet 9a89f25021f7427c815f86026e11ffa8 (DEFAULT_TABLE table=test-table [id=dc87aff5877b41d48bb6247ca836215f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:03.775353 10134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a89f25021f7427c815f86026e11ffa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:03.776535 10371 tablet_bootstrap.cc:492] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b: Bootstrap starting.
I20251214 03:03:03.776760 10276 tablet_service.cc:1505] Processing CreateTablet for tablet 9a89f25021f7427c815f86026e11ffa8 (DEFAULT_TABLE table=test-table [id=dc87aff5877b41d48bb6247ca836215f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:03.776994 10276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a89f25021f7427c815f86026e11ffa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:03.777418 10372 tablet_bootstrap.cc:492] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260: Bootstrap starting.
I20251214 03:03:03.778357 10372 tablet_bootstrap.cc:654] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:03.779206 10370 tablet_bootstrap.cc:492] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0: Bootstrap starting.
I20251214 03:03:03.780154 10370 tablet_bootstrap.cc:654] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:03.780328 10395 ts_tablet_manager.cc:1929] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:03:03.780432 10395 log.cc:1199] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/09d0771515c64e79bc24984a21435587
I20251214 03:03:03.780709 10395 ts_tablet_manager.cc:1950] T 09d0771515c64e79bc24984a21435587 P bae10355c88b4e40a02e6016c9b2d260: Deleting consensus metadata
I20251214 03:03:03.781731 10370 tablet_bootstrap.cc:492] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0: No bootstrap required, opened a new log
I20251214 03:03:03.781814 10370 ts_tablet_manager.cc:1403] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251214 03:03:03.781859 10371 tablet_bootstrap.cc:654] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:03.782234 10370 raft_consensus.cc:359] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.782320 10370 raft_consensus.cc:385] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:03.782341 10370 raft_consensus.cc:740] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a9b80de17e4062846ac8b3f874d6b0, State: Initialized, Role: FOLLOWER
I20251214 03:03:03.782471 10370 consensus_queue.cc:260] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.782727 10370 ts_tablet_manager.cc:1434] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:03:03.782759 10372 tablet_bootstrap.cc:492] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260: No bootstrap required, opened a new log
I20251214 03:03:03.782833 10372 ts_tablet_manager.cc:1403] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251214 03:03:03.783089 10371 tablet_bootstrap.cc:492] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b: No bootstrap required, opened a new log
I20251214 03:03:03.783185 10371 ts_tablet_manager.cc:1403] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251214 03:03:03.783172 9837 catalog_manager.cc:4985] TS bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519): tablet 09d0771515c64e79bc24984a21435587 (table test-table [id=3fa66c522ffa40538544719cf34ca8fd]) successfully deleted
I20251214 03:03:03.783268 10372 raft_consensus.cc:359] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.783370 10372 raft_consensus.cc:385] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:03.783404 10372 raft_consensus.cc:740] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae10355c88b4e40a02e6016c9b2d260, State: Initialized, Role: FOLLOWER
I20251214 03:03:03.783521 10372 consensus_queue.cc:260] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.783627 10371 raft_consensus.cc:359] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.783725 10371 raft_consensus.cc:385] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:03.783749 10372 ts_tablet_manager.cc:1434] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 03:03:03.783759 10371 raft_consensus.cc:740] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152aa837d18449d891ba835a3cc0420b, State: Initialized, Role: FOLLOWER
I20251214 03:03:03.783883 10371 consensus_queue.cc:260] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.784133 10371 ts_tablet_manager.cc:1434] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:03:03.784922 10394 ts_tablet_manager.cc:1929] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:03:03.785006 10394 log.cc:1199] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/09d0771515c64e79bc24984a21435587
I20251214 03:03:03.785344 10394 ts_tablet_manager.cc:1950] T 09d0771515c64e79bc24984a21435587 P a9a9b80de17e4062846ac8b3f874d6b0: Deleting consensus metadata
I20251214 03:03:03.786566 9839 catalog_manager.cc:4985] TS a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545): tablet 09d0771515c64e79bc24984a21435587 (table test-table [id=3fa66c522ffa40538544719cf34ca8fd]) successfully deleted
I20251214 03:03:03.838708 10377 raft_consensus.cc:493] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:03:03.838840 10377 raft_consensus.cc:515] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.839902 10377 leader_election.cc:290] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591), a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545)
I20251214 03:03:03.843289 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.840094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:34591 (local address 127.1.157.194:59121)
1214 03:03:03.840419 (+ 325us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.840433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.840599 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.840871 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.840874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.840894 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:03.841117 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.841123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.841880 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.841884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.842860 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.842870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.842992 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.843015 (+ 23us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.843063 (+ 48us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.843126 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"spinlock_wait_cycles":5248,"thread_start_us":70,"threads_started":1}
I20251214 03:03:03.843739 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.840258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:59121 (local address 127.1.157.193:34591)
1214 03:03:03.840373 (+ 115us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.840377 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.840498 (+ 121us) server_negotiation.cc:408] Connection header received
1214 03:03:03.840741 (+ 243us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.840744 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.840788 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.840850 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:03.841255 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.841777 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.842996 (+ 1219us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.843480 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.843520 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.843571 (+ 51us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.843632 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251214 03:03:03.843997 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.840547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:33387 (local address 127.1.157.195:41545)
1214 03:03:03.840663 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.840667 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.840682 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:03:03.840734 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.840738 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.840788 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.840864 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:03.841778 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.842279 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.843123 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.843575 (+ 452us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.843608 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.843801 (+ 193us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.843883 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251214 03:03:03.844219 10012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a89f25021f7427c815f86026e11ffa8" candidate_uuid: "bae10355c88b4e40a02e6016c9b2d260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152aa837d18449d891ba835a3cc0420b" is_pre_election: true
I20251214 03:03:03.844373 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.840155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41545 (local address 127.1.157.194:33387)
1214 03:03:03.840494 (+ 339us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.840506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.840599 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.840966 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.840970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.841345 (+ 375us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:03.841640 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.841650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.842384 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.842388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.842988 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.842996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.843658 (+ 662us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.843675 (+ 17us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.844177 (+ 502us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.844240 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"spinlock_wait_cycles":8576,"thread_start_us":116,"threads_started":1}
I20251214 03:03:03.844396 10012 raft_consensus.cc:2468] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae10355c88b4e40a02e6016c9b2d260 in term 0.
I20251214 03:03:03.844805 10296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a89f25021f7427c815f86026e11ffa8" candidate_uuid: "bae10355c88b4e40a02e6016c9b2d260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" is_pre_election: true
I20251214 03:03:03.844929 10296 raft_consensus.cc:2468] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae10355c88b4e40a02e6016c9b2d260 in term 0.
I20251214 03:03:03.844947 10085 leader_election.cc:304] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152aa837d18449d891ba835a3cc0420b, bae10355c88b4e40a02e6016c9b2d260; no voters:
I20251214 03:03:03.845219 10377 raft_consensus.cc:2804] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:03:03.845283 10377 raft_consensus.cc:493] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:03:03.845321 10377 raft_consensus.cc:3060] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:03.846220 10377 raft_consensus.cc:515] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.846606 10377 leader_election.cc:290] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [CANDIDATE]: Term 1 election: Requested vote from peers 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591), a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545)
I20251214 03:03:03.846940 10012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a89f25021f7427c815f86026e11ffa8" candidate_uuid: "bae10355c88b4e40a02e6016c9b2d260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152aa837d18449d891ba835a3cc0420b"
I20251214 03:03:03.847050 10012 raft_consensus.cc:3060] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:03.847069 10296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a89f25021f7427c815f86026e11ffa8" candidate_uuid: "bae10355c88b4e40a02e6016c9b2d260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a9b80de17e4062846ac8b3f874d6b0"
I20251214 03:03:03.847186 10296 raft_consensus.cc:3060] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:03.847942 10012 raft_consensus.cc:2468] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae10355c88b4e40a02e6016c9b2d260 in term 1.
I20251214 03:03:03.848181 10296 raft_consensus.cc:2468] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae10355c88b4e40a02e6016c9b2d260 in term 1.
I20251214 03:03:03.848243 10085 leader_election.cc:304] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152aa837d18449d891ba835a3cc0420b, bae10355c88b4e40a02e6016c9b2d260; no voters:
I20251214 03:03:03.848390 10377 raft_consensus.cc:2804] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:03.848577 10377 raft_consensus.cc:697] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 1 LEADER]: Becoming Leader. State: Replica: bae10355c88b4e40a02e6016c9b2d260, State: Running, Role: LEADER
I20251214 03:03:03.848805 10377 consensus_queue.cc:237] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.851037 9847 catalog_manager.cc:5654] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 reported cstate change: term changed from 0 to 1, leader changed from <none> to bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194). New cstate: current_term: 1 leader_uuid: "bae10355c88b4e40a02e6016c9b2d260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } health_report { overall_health: UNKNOWN } } }
Dec 14 03:03:03 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681382, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.254@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:03:03.894481 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.886853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.1.157.254:36667)
1214 03:03:03.887090 (+ 237us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:03.887094 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:03.887107 (+ 13us) server_negotiation.cc:408] Connection header received
1214 03:03:03.887204 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:03.887208 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:03.887248 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:03.887313 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:03.888018 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.888536 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.889079 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.889264 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.891531 (+ 2267us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:03.891546 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:03.891548 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:03.891569 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:03.893036 (+ 1467us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.893484 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.893487 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.893488 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.893527 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:03.893711 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:03.893715 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:03.893717 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:03.893847 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:03.893927 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:03.894181 (+ 254us) server_negotiation.cc:300] Negotiation successful
1214 03:03:03.894312 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":169,"thread_start_us":113,"threads_started":1}
I20251214 03:03:03.894481 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:03.886775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:36667 (local address 127.0.0.1:42764)
1214 03:03:03.886994 (+ 219us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:03.887011 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:03.887119 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:03.887342 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:03.887345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:03.887722 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:03.887905 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.887911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.888631 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:03.888634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:03.888987 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:03.888994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:03.889089 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:03.889816 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:03.889830 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:03.891409 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:03.893145 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:03.893152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:03.893157 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:03.893390 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:03.893598 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:03.893601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:03.893603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:03.893644 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:03.893935 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:03.893942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:03.894029 (+ 87us) client_negotiation.cc:769] Sending connection context
1214 03:03:03.894118 (+ 89us) client_negotiation.cc:240] Negotiation successful
1214 03:03:03.894257 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20251214 03:03:03.897984 9847 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:42764:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251214 03:03:03.898644 9847 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:03:03.903474 9992 tablet_service.cc:1505] Processing CreateTablet for tablet e72aac65a7724865a50a71a8ff58f565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4dfd30c3f9924913958f2e841e756d49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251214 03:03:03.903643 10134 tablet_service.cc:1505] Processing CreateTablet for tablet e72aac65a7724865a50a71a8ff58f565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4dfd30c3f9924913958f2e841e756d49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251214 03:03:03.903841 9992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72aac65a7724865a50a71a8ff58f565. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:03.903785 10276 tablet_service.cc:1505] Processing CreateTablet for tablet e72aac65a7724865a50a71a8ff58f565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4dfd30c3f9924913958f2e841e756d49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251214 03:03:03.904023 10134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72aac65a7724865a50a71a8ff58f565. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:03.904089 10276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72aac65a7724865a50a71a8ff58f565. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:03.906533 10371 tablet_bootstrap.cc:492] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: Bootstrap starting.
I20251214 03:03:03.906584 10370 tablet_bootstrap.cc:492] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: Bootstrap starting.
I20251214 03:03:03.906713 10372 tablet_bootstrap.cc:492] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: Bootstrap starting.
I20251214 03:03:03.907485 10370 tablet_bootstrap.cc:654] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:03.907485 10371 tablet_bootstrap.cc:654] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:03.907780 10372 tablet_bootstrap.cc:654] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:03.909251 10370 tablet_bootstrap.cc:492] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: No bootstrap required, opened a new log
I20251214 03:03:03.909355 10370 ts_tablet_manager.cc:1403] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251214 03:03:03.909494 10371 tablet_bootstrap.cc:492] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: No bootstrap required, opened a new log
I20251214 03:03:03.909564 10371 ts_tablet_manager.cc:1403] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251214 03:03:03.909976 10371 raft_consensus.cc:359] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.909967 10370 raft_consensus.cc:359] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.910077 10371 raft_consensus.cc:385] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:03.910118 10371 raft_consensus.cc:740] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152aa837d18449d891ba835a3cc0420b, State: Initialized, Role: FOLLOWER
I20251214 03:03:03.910076 10370 raft_consensus.cc:385] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:03.910161 10370 raft_consensus.cc:740] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a9b80de17e4062846ac8b3f874d6b0, State: Initialized, Role: FOLLOWER
I20251214 03:03:03.910238 10371 consensus_queue.cc:260] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.910313 10370 consensus_queue.cc:260] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.910504 10371 ts_tablet_manager.cc:1434] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251214 03:03:03.910591 10370 ts_tablet_manager.cc:1434] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:03:03.910504 10391 tablet_replica.cc:442] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } }
I20251214 03:03:03.910709 10391 tablet_replica.cc:445] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:03:03.910959 10378 tablet_replica.cc:442] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } }
I20251214 03:03:03.911072 10378 tablet_replica.cc:445] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:03:03.911487 10372 tablet_bootstrap.cc:492] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: No bootstrap required, opened a new log
I20251214 03:03:03.911563 10372 ts_tablet_manager.cc:1403] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251214 03:03:03.912003 10372 raft_consensus.cc:359] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.912110 10372 raft_consensus.cc:385] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:03.912146 10372 raft_consensus.cc:740] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae10355c88b4e40a02e6016c9b2d260, State: Initialized, Role: FOLLOWER
I20251214 03:03:03.912261 10372 consensus_queue.cc:260] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:03.912956 10377 tablet_replica.cc:442] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } }
I20251214 03:03:03.913064 10377 tablet_replica.cc:445] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:03:03.913292 10372 ts_tablet_manager.cc:1434] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 03:03:04.111728 10378 raft_consensus.cc:493] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:03:04.111874 10378 raft_consensus.cc:515] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.112921 10378 leader_election.cc:290] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591), bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519)
I20251214 03:03:04.116364 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.113556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:39519 (local address 127.1.157.195:59617)
1214 03:03:04.113819 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:04.113831 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:04.113948 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:04.114225 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:04.114228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:04.114245 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:04.114520 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.114527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.115380 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.115383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:04.115960 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.115967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.116083 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.116107 (+ 24us) client_negotiation.cc:769] Sending connection context
1214 03:03:04.116157 (+ 50us) client_negotiation.cc:240] Negotiation successful
1214 03:03:04.116213 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20251214 03:03:04.116786 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.113663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:59617 (local address 127.1.157.194:39519)
1214 03:03:04.113825 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.113829 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.113911 (+ 82us) server_negotiation.cc:408] Connection header received
1214 03:03:04.114081 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.114085 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.114130 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.114201 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:04.114640 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.115279 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.116083 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.116533 (+ 450us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.116573 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.116623 (+ 50us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.116683 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20251214 03:03:04.117045 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.113146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:34591 (local address 127.1.157.195:43803)
1214 03:03:04.113424 (+ 278us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:04.113439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:04.113550 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:04.113956 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:04.113959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:04.113980 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:04.114263 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.114273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.115116 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.115120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:04.116041 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.116049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.116764 (+ 715us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.116780 (+ 16us) client_negotiation.cc:769] Sending connection context
1214 03:03:04.116855 (+ 75us) client_negotiation.cc:240] Negotiation successful
1214 03:03:04.116913 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20251214 03:03:04.117090 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.113259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:43803 (local address 127.1.157.193:34591)
1214 03:03:04.113409 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.113413 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.113513 (+ 100us) server_negotiation.cc:408] Connection header received
1214 03:03:04.113705 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.113709 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.113764 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.113813 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:04.114371 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.115016 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.116172 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.116654 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.116691 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.116903 (+ 212us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.116987 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251214 03:03:04.117199 10154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e72aac65a7724865a50a71a8ff58f565" candidate_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae10355c88b4e40a02e6016c9b2d260" is_pre_election: true
I20251214 03:03:04.117331 10154 raft_consensus.cc:2468] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9a9b80de17e4062846ac8b3f874d6b0 in term 0.
I20251214 03:03:04.117489 10012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e72aac65a7724865a50a71a8ff58f565" candidate_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152aa837d18449d891ba835a3cc0420b" is_pre_election: true
I20251214 03:03:04.117612 10012 raft_consensus.cc:2468] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9a9b80de17e4062846ac8b3f874d6b0 in term 0.
I20251214 03:03:04.117847 10227 leader_election.cc:304] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9a9b80de17e4062846ac8b3f874d6b0, bae10355c88b4e40a02e6016c9b2d260; no voters:
I20251214 03:03:04.118064 10378 raft_consensus.cc:2804] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:03:04.118125 10378 raft_consensus.cc:493] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:03:04.118170 10378 raft_consensus.cc:3060] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:04.119045 10378 raft_consensus.cc:515] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.119385 10378 leader_election.cc:290] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [CANDIDATE]: Term 1 election: Requested vote from peers 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591), bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519)
I20251214 03:03:04.119769 10012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e72aac65a7724865a50a71a8ff58f565" candidate_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152aa837d18449d891ba835a3cc0420b"
I20251214 03:03:04.119776 10154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e72aac65a7724865a50a71a8ff58f565" candidate_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae10355c88b4e40a02e6016c9b2d260"
I20251214 03:03:04.119880 10012 raft_consensus.cc:3060] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:04.119892 10154 raft_consensus.cc:3060] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:04.120712 10012 raft_consensus.cc:2468] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9a9b80de17e4062846ac8b3f874d6b0 in term 1.
I20251214 03:03:04.120908 10154 raft_consensus.cc:2468] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9a9b80de17e4062846ac8b3f874d6b0 in term 1.
I20251214 03:03:04.121002 10227 leader_election.cc:304] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152aa837d18449d891ba835a3cc0420b, a9a9b80de17e4062846ac8b3f874d6b0; no voters:
I20251214 03:03:04.121306 10378 raft_consensus.cc:2804] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:04.121487 10378 raft_consensus.cc:697] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 LEADER]: Becoming Leader. State: Replica: a9a9b80de17e4062846ac8b3f874d6b0, State: Running, Role: LEADER
I20251214 03:03:04.121742 10378 consensus_queue.cc:237] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.122890 10416 tablet_replica.cc:442] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: TxnStatusTablet state changed. Reason: New leader a9a9b80de17e4062846ac8b3f874d6b0. Latest consensus state: current_term: 1 leader_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } }
I20251214 03:03:04.122975 10416 tablet_replica.cc:445] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: This TxnStatusTablet replica's current role is: LEADER
I20251214 03:03:04.123257 10418 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251214 03:03:04.123380 10418 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251214 03:03:04.124410 9853 catalog_manager.cc:5654] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195). New cstate: current_term: 1 leader_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } health_report { overall_health: HEALTHY } } }
I20251214 03:03:04.150398 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.147199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.1.157.195:41545)
1214 03:03:04.147311 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.147314 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.147389 (+ 75us) server_negotiation.cc:408] Connection header received
1214 03:03:04.147581 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.147584 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.147636 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.147712 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:04.148091 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.148575 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.149487 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.149618 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.149665 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.150068 (+ 403us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.150150 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.150248 (+ 98us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.150293 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251214 03:03:04.150450 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.147073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41545 (local address 127.0.0.1:50866)
1214 03:03:04.147322 (+ 249us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:04.147336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:04.147462 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:04.147736 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:04.147739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:04.147747 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:03:04.147999 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.148006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.148671 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.148675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:04.149382 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.149391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.149498 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.149545 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.150157 (+ 612us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.150169 (+ 12us) client_negotiation.cc:769] Sending connection context
1214 03:03:04.150242 (+ 73us) client_negotiation.cc:240] Negotiation successful
1214 03:03:04.150316 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20251214 03:03:04.153533 10154 raft_consensus.cc:1275] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Refusing update from remote peer a9a9b80de17e4062846ac8b3f874d6b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:04.153708 10012 raft_consensus.cc:1275] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Refusing update from remote peer a9a9b80de17e4062846ac8b3f874d6b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:04.154057 10416 consensus_queue.cc:1048] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:04.154278 10378 consensus_queue.cc:1048] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:04.155870 10391 tablet_replica.cc:442] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: TxnStatusTablet state changed. Reason: New leader a9a9b80de17e4062846ac8b3f874d6b0. Latest consensus state: current_term: 1 leader_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } }
I20251214 03:03:04.155972 10391 tablet_replica.cc:445] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:03:04.155974 10377 tablet_replica.cc:442] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: TxnStatusTablet state changed. Reason: New leader a9a9b80de17e4062846ac8b3f874d6b0. Latest consensus state: current_term: 1 leader_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } }
I20251214 03:03:04.156077 10377 tablet_replica.cc:445] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:03:04.157720 10378 tablet_replica.cc:442] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } }
I20251214 03:03:04.157814 10378 tablet_replica.cc:445] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: This TxnStatusTablet replica's current role is: LEADER
I20251214 03:03:04.157891 10385 mvcc.cc:204] Tried to move back new op lower bound from 7232230949486960640 to 7232230949365137408. Current Snapshot: MvccSnapshot[applied={T|T < 7232230949486960640}]
I20251214 03:03:04.158404 10378 tablet_replica.cc:442] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } }
I20251214 03:03:04.158490 10378 tablet_replica.cc:445] T e72aac65a7724865a50a71a8ff58f565 P a9a9b80de17e4062846ac8b3f874d6b0: This TxnStatusTablet replica's current role is: LEADER
I20251214 03:03:04.159724 10391 tablet_replica.cc:442] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } }
I20251214 03:03:04.159823 10391 tablet_replica.cc:445] T e72aac65a7724865a50a71a8ff58f565 P 152aa837d18449d891ba835a3cc0420b: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:03:04.159902 10384 mvcc.cc:204] Tried to move back new op lower bound from 7232230949486960640 to 7232230949365137408. Current Snapshot: MvccSnapshot[applied={T|T < 7232230949486960640 or (T in {7232230949486960640})}]
I20251214 03:03:04.161973 10377 tablet_replica.cc:442] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } } }
I20251214 03:03:04.162065 10377 tablet_replica.cc:445] T e72aac65a7724865a50a71a8ff58f565 P bae10355c88b4e40a02e6016c9b2d260: This TxnStatusTablet replica's current role is: FOLLOWER
I20251214 03:03:04.166108 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.162483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.1.157.194:39519)
1214 03:03:04.162818 (+ 335us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.162823 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.162835 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:03:04.162905 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.162908 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.162966 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.163050 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:04.163541 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.164278 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.164874 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.165066 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.165123 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.165714 (+ 591us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.165819 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.165935 (+ 116us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.165986 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20251214 03:03:04.175777 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.173033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41545 (local address 127.1.157.194:40345)
1214 03:03:04.173336 (+ 303us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:04.173350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:04.173453 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:04.173688 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:04.173691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:04.173698 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:03:04.173916 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.173922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.174577 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.174580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:04.175009 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.175016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.175125 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.175171 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.175500 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.175504 (+ 4us) client_negotiation.cc:769] Sending connection context
1214 03:03:04.175559 (+ 55us) client_negotiation.cc:240] Negotiation successful
1214 03:03:04.175630 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
I20251214 03:03:04.175762 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.173135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:40345 (local address 127.1.157.195:41545)
1214 03:03:04.173312 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.173317 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.173413 (+ 96us) server_negotiation.cc:408] Connection header received
1214 03:03:04.173552 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.173556 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.173607 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.173671 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:04.174014 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.174476 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.175112 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.175302 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.175342 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.175408 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.175468 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.175576 (+ 108us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.175621 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251214 03:03:04.184351 10296 raft_consensus.cc:1275] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Refusing update from remote peer bae10355c88b4e40a02e6016c9b2d260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:04.184392 10012 raft_consensus.cc:1275] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Refusing update from remote peer bae10355c88b4e40a02e6016c9b2d260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:04.184985 10402 consensus_queue.cc:1048] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:04.185154 10377 consensus_queue.cc:1048] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:04.213973 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.211067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:59931 (local address 127.1.157.194:39519)
1214 03:03:04.211187 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.211191 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.211368 (+ 177us) server_negotiation.cc:408] Connection header received
1214 03:03:04.211509 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.211512 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.211567 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.211629 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:04.212107 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.212609 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.213342 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.213518 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.213563 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.213637 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.213699 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.213800 (+ 101us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.213844 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251214 03:03:04.213987 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.210966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:39519 (local address 127.1.157.195:59931)
1214 03:03:04.211280 (+ 314us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:04.211296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:04.211400 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:04.211646 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:04.211649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:04.211658 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:03:04.211992 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.212000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.212713 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.212716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:04.213246 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.213257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.213357 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.213403 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.213717 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.213724 (+ 7us) client_negotiation.cc:769] Sending connection context
1214 03:03:04.213781 (+ 57us) client_negotiation.cc:240] Negotiation successful
1214 03:03:04.213852 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":100,"threads_started":1}
I20251214 03:03:04.241969 9847 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42754:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:04.242159 9847 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42754:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:04.244441 9847 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3: Sending DeleteTablet for 3 replicas of tablet 9a89f25021f7427c815f86026e11ffa8
I20251214 03:03:04.245118 10276 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a89f25021f7427c815f86026e11ffa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:04 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:50856
I20251214 03:03:04.245131 10134 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a89f25021f7427c815f86026e11ffa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:04 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:38228
I20251214 03:03:04.245342 10394 tablet_replica.cc:333] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0: stopping tablet replica
I20251214 03:03:04.245453 10394 raft_consensus.cc:2243] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:04.245559 10394 raft_consensus.cc:2272] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:04.245555 9992 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a89f25021f7427c815f86026e11ffa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:04 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:55440
I20251214 03:03:04.245714 10395 tablet_replica.cc:333] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260: stopping tablet replica
I20251214 03:03:04.245729 10396 tablet_replica.cc:333] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b: stopping tablet replica
I20251214 03:03:04.245812 10395 raft_consensus.cc:2243] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:03:04.245846 10396 raft_consensus.cc:2243] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:04.245949 10396 raft_consensus.cc:2272] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:04.246081 10395 raft_consensus.cc:2272] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:04.246903 10395 ts_tablet_manager.cc:1916] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:04.247207 10396 ts_tablet_manager.cc:1916] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:04.249650 10395 ts_tablet_manager.cc:1929] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:03:04.249722 10395 log.cc:1199] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/9a89f25021f7427c815f86026e11ffa8
I20251214 03:03:04.249732 10396 ts_tablet_manager.cc:1929] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:03:04.249796 10396 log.cc:1199] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/wal/wals/9a89f25021f7427c815f86026e11ffa8
I20251214 03:03:04.249969 10395 ts_tablet_manager.cc:1950] T 9a89f25021f7427c815f86026e11ffa8 P bae10355c88b4e40a02e6016c9b2d260: Deleting consensus metadata
I20251214 03:03:04.250033 10396 ts_tablet_manager.cc:1950] T 9a89f25021f7427c815f86026e11ffa8 P 152aa837d18449d891ba835a3cc0420b: Deleting consensus metadata
I20251214 03:03:04.250873 10394 ts_tablet_manager.cc:1916] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:04.251112 9837 catalog_manager.cc:4985] TS 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591): tablet 9a89f25021f7427c815f86026e11ffa8 (table test-table [id=dc87aff5877b41d48bb6247ca836215f]) successfully deleted
I20251214 03:03:04.253551 10394 ts_tablet_manager.cc:1929] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:03:04.253713 10394 log.cc:1199] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/9a89f25021f7427c815f86026e11ffa8
I20251214 03:03:04.254029 10394 ts_tablet_manager.cc:1950] T 9a89f25021f7427c815f86026e11ffa8 P a9a9b80de17e4062846ac8b3f874d6b0: Deleting consensus metadata
I20251214 03:03:04.255064 9839 catalog_manager.cc:4985] TS a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545): tablet 9a89f25021f7427c815f86026e11ffa8 (table test-table [id=dc87aff5877b41d48bb6247ca836215f]) successfully deleted
I20251214 03:03:04.258003 9837 catalog_manager.cc:4985] TS bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519): tablet 9a89f25021f7427c815f86026e11ffa8 (table test-table [id=dc87aff5877b41d48bb6247ca836215f]) successfully deleted
Dec 14 03:03:04 dist-test-slave-k7z7 krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 14 03:03:04 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:04 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
W20251214 03:03:04.285995 10410 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:04.275949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.1.157.254:36667)
1214 03:03:04.276086 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.276091 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.276272 (+ 181us) server_negotiation.cc:408] Connection header received
1214 03:03:04.276401 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.276405 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.276464 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.276560 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:04.277630 (+ 1070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.278452 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.279346 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.279562 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.285863 (+ 6301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42772: BlockingRecv error: recv got EOF from 127.0.0.1:42772 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Dec 14 03:03:04 dist-test-slave-k7z7 krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM
I20251214 03:03:04.296969 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.287482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.1.157.254:36667)
1214 03:03:04.287603 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.287607 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.287785 (+ 178us) server_negotiation.cc:408] Connection header received
1214 03:03:04.287910 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.287913 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.287967 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.288052 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:04.288919 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.289800 (+ 881us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.290667 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.290862 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.293557 (+ 2695us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:04.293583 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:04.293587 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:04.293624 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:04.295346 (+ 1722us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:04.295795 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:04.295799 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:04.295801 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:04.295857 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:04.296082 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:04.296085 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:04.296087 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:04.296301 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:04.296443 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.296663 (+ 220us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.296800 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251214 03:03:04.299866 9847 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:03:04.300370 9847 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:03:04.305701 9992 tablet_service.cc:1505] Processing CreateTablet for tablet 4a8dc12d4c5f4dae990d01852f0a6d06 (DEFAULT_TABLE table=test-table [id=935c45f7d2a84b718b801adb11db2462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:04.306007 9992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8dc12d4c5f4dae990d01852f0a6d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:04.308311 10371 tablet_bootstrap.cc:492] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b: Bootstrap starting.
I20251214 03:03:04.309309 10371 tablet_bootstrap.cc:654] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:04.310376 10276 tablet_service.cc:1505] Processing CreateTablet for tablet 4a8dc12d4c5f4dae990d01852f0a6d06 (DEFAULT_TABLE table=test-table [id=935c45f7d2a84b718b801adb11db2462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:04.310670 10276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8dc12d4c5f4dae990d01852f0a6d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:04.312965 10370 tablet_bootstrap.cc:492] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0: Bootstrap starting.
I20251214 03:03:04.313808 10134 tablet_service.cc:1505] Processing CreateTablet for tablet 4a8dc12d4c5f4dae990d01852f0a6d06 (DEFAULT_TABLE table=test-table [id=935c45f7d2a84b718b801adb11db2462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:04.314075 10134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8dc12d4c5f4dae990d01852f0a6d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:04.314142 10371 tablet_bootstrap.cc:492] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b: No bootstrap required, opened a new log
I20251214 03:03:04.314249 10371 ts_tablet_manager.cc:1403] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20251214 03:03:04.314678 10371 raft_consensus.cc:359] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.314781 10371 raft_consensus.cc:385] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:04.314807 10371 raft_consensus.cc:740] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152aa837d18449d891ba835a3cc0420b, State: Initialized, Role: FOLLOWER
I20251214 03:03:04.314929 10371 consensus_queue.cc:260] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.315181 10371 ts_tablet_manager.cc:1434] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251214 03:03:04.315806 10370 tablet_bootstrap.cc:654] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:04.318609 10372 tablet_bootstrap.cc:492] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260: Bootstrap starting.
I20251214 03:03:04.319366 10370 tablet_bootstrap.cc:492] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0: No bootstrap required, opened a new log
I20251214 03:03:04.319446 10370 ts_tablet_manager.cc:1403] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251214 03:03:04.319567 10372 tablet_bootstrap.cc:654] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:04.319926 10370 raft_consensus.cc:359] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.320034 10370 raft_consensus.cc:385] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:04.320063 10370 raft_consensus.cc:740] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a9b80de17e4062846ac8b3f874d6b0, State: Initialized, Role: FOLLOWER
I20251214 03:03:04.320183 10370 consensus_queue.cc:260] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.320436 10370 ts_tablet_manager.cc:1434] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 03:03:04.320662 10372 tablet_bootstrap.cc:492] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260: No bootstrap required, opened a new log
I20251214 03:03:04.320736 10372 ts_tablet_manager.cc:1403] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:04.321081 10372 raft_consensus.cc:359] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.321151 10372 raft_consensus.cc:385] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:04.321201 10372 raft_consensus.cc:740] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae10355c88b4e40a02e6016c9b2d260, State: Initialized, Role: FOLLOWER
I20251214 03:03:04.321327 10372 consensus_queue.cc:260] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.321540 10372 ts_tablet_manager.cc:1434] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 03:03:04.529247 10426 raft_consensus.cc:493] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:03:04.529381 10426 raft_consensus.cc:515] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.529778 10426 leader_election.cc:290] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591), bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519)
I20251214 03:03:04.530288 10011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8dc12d4c5f4dae990d01852f0a6d06" candidate_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152aa837d18449d891ba835a3cc0420b" is_pre_election: true
I20251214 03:03:04.530275 10154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8dc12d4c5f4dae990d01852f0a6d06" candidate_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae10355c88b4e40a02e6016c9b2d260" is_pre_election: true
I20251214 03:03:04.530422 10154 raft_consensus.cc:2468] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9a9b80de17e4062846ac8b3f874d6b0 in term 0.
I20251214 03:03:04.530422 10011 raft_consensus.cc:2468] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9a9b80de17e4062846ac8b3f874d6b0 in term 0.
I20251214 03:03:04.530699 10227 leader_election.cc:304] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9a9b80de17e4062846ac8b3f874d6b0, bae10355c88b4e40a02e6016c9b2d260; no voters:
I20251214 03:03:04.530853 10426 raft_consensus.cc:2804] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:03:04.530910 10426 raft_consensus.cc:493] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:03:04.530947 10426 raft_consensus.cc:3060] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:04.531853 10426 raft_consensus.cc:515] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.532150 10426 leader_election.cc:290] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [CANDIDATE]: Term 1 election: Requested vote from peers 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591), bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519)
I20251214 03:03:04.532466 10011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8dc12d4c5f4dae990d01852f0a6d06" candidate_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152aa837d18449d891ba835a3cc0420b"
I20251214 03:03:04.532526 10154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8dc12d4c5f4dae990d01852f0a6d06" candidate_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae10355c88b4e40a02e6016c9b2d260"
I20251214 03:03:04.532572 10011 raft_consensus.cc:3060] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:04.532630 10154 raft_consensus.cc:3060] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:04.533489 10011 raft_consensus.cc:2468] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9a9b80de17e4062846ac8b3f874d6b0 in term 1.
I20251214 03:03:04.533500 10154 raft_consensus.cc:2468] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9a9b80de17e4062846ac8b3f874d6b0 in term 1.
I20251214 03:03:04.533763 10227 leader_election.cc:304] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152aa837d18449d891ba835a3cc0420b, a9a9b80de17e4062846ac8b3f874d6b0; no voters:
I20251214 03:03:04.533912 10426 raft_consensus.cc:2804] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:04.533998 10426 raft_consensus.cc:697] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 LEADER]: Becoming Leader. State: Replica: a9a9b80de17e4062846ac8b3f874d6b0, State: Running, Role: LEADER
I20251214 03:03:04.534132 10426 consensus_queue.cc:237] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.535676 9852 catalog_manager.cc:5654] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195). New cstate: current_term: 1 leader_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } health_report { overall_health: HEALTHY } } }
I20251214 03:03:04.545498 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.543142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.1.157.195:41545)
1214 03:03:04.543260 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.543265 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.543278 (+ 13us) server_negotiation.cc:408] Connection header received
1214 03:03:04.543386 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.543389 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.543433 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.543505 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:04.543806 (+ 301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.544258 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.544868 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.545027 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.545068 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.545130 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.545228 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.545322 (+ 94us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.545367 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251214 03:03:04.547390 10154 raft_consensus.cc:1275] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Refusing update from remote peer a9a9b80de17e4062846ac8b3f874d6b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:04.547432 10011 raft_consensus.cc:1275] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Refusing update from remote peer a9a9b80de17e4062846ac8b3f874d6b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:04.547768 10378 consensus_queue.cc:1048] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:04.547892 10426 consensus_queue.cc:1048] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:04.556077 9852 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:04.556259 9852 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:04.558240 9852 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3: Sending DeleteTablet for 3 replicas of tablet 4a8dc12d4c5f4dae990d01852f0a6d06
I20251214 03:03:04.558739 9992 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a8dc12d4c5f4dae990d01852f0a6d06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:04 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:55440
I20251214 03:03:04.558888 10276 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a8dc12d4c5f4dae990d01852f0a6d06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:04 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:50856
I20251214 03:03:04.558926 10134 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a8dc12d4c5f4dae990d01852f0a6d06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:04 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:38228
I20251214 03:03:04.559058 10394 tablet_replica.cc:333] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0: stopping tablet replica
I20251214 03:03:04.559091 10396 tablet_replica.cc:333] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b: stopping tablet replica
I20251214 03:03:04.559167 10394 raft_consensus.cc:2243] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:03:04.559203 10396 raft_consensus.cc:2243] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:04.559301 10396 raft_consensus.cc:2272] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:04.559442 10395 tablet_replica.cc:333] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260: stopping tablet replica
I20251214 03:03:04.559446 10394 raft_consensus.cc:2272] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:04.559549 10395 raft_consensus.cc:2243] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:04.559656 10395 raft_consensus.cc:2272] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:04.560158 10394 ts_tablet_manager.cc:1916] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:04.560420 10395 ts_tablet_manager.cc:1916] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:04.560572 10396 ts_tablet_manager.cc:1916] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:04.562651 10394 ts_tablet_manager.cc:1929] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:03:04.562661 10395 ts_tablet_manager.cc:1929] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:03:04.562661 10396 ts_tablet_manager.cc:1929] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:03:04.562726 10395 log.cc:1199] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/4a8dc12d4c5f4dae990d01852f0a6d06
I20251214 03:03:04.562726 10396 log.cc:1199] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-0/wal/wals/4a8dc12d4c5f4dae990d01852f0a6d06
I20251214 03:03:04.562840 10394 log.cc:1199] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/4a8dc12d4c5f4dae990d01852f0a6d06
I20251214 03:03:04.562970 10396 ts_tablet_manager.cc:1950] T 4a8dc12d4c5f4dae990d01852f0a6d06 P 152aa837d18449d891ba835a3cc0420b: Deleting consensus metadata
I20251214 03:03:04.562970 10395 ts_tablet_manager.cc:1950] T 4a8dc12d4c5f4dae990d01852f0a6d06 P bae10355c88b4e40a02e6016c9b2d260: Deleting consensus metadata
I20251214 03:03:04.563184 10394 ts_tablet_manager.cc:1950] T 4a8dc12d4c5f4dae990d01852f0a6d06 P a9a9b80de17e4062846ac8b3f874d6b0: Deleting consensus metadata
I20251214 03:03:04.563676 9837 catalog_manager.cc:4985] TS 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591): tablet 4a8dc12d4c5f4dae990d01852f0a6d06 (table test-table [id=935c45f7d2a84b718b801adb11db2462]) successfully deleted
I20251214 03:03:04.563843 9837 catalog_manager.cc:4985] TS bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519): tablet 4a8dc12d4c5f4dae990d01852f0a6d06 (table test-table [id=935c45f7d2a84b718b801adb11db2462]) successfully deleted
I20251214 03:03:04.563868 9839 catalog_manager.cc:4985] TS a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545): tablet 4a8dc12d4c5f4dae990d01852f0a6d06 (table test-table [id=935c45f7d2a84b718b801adb11db2462]) successfully deleted
I20251214 03:03:04.564889 9852 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:03:04.565402 9852 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:03:04.581189 9992 tablet_service.cc:1505] Processing CreateTablet for tablet 0442b9d36d63446ba21ad3816d7a91bd (DEFAULT_TABLE table=test-table [id=4d082de0c00e4cba9da80a3d390f559d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:04.581391 10134 tablet_service.cc:1505] Processing CreateTablet for tablet 0442b9d36d63446ba21ad3816d7a91bd (DEFAULT_TABLE table=test-table [id=4d082de0c00e4cba9da80a3d390f559d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:04.581501 9992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0442b9d36d63446ba21ad3816d7a91bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:04.581646 10134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0442b9d36d63446ba21ad3816d7a91bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:04.583109 10371 tablet_bootstrap.cc:492] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b: Bootstrap starting.
I20251214 03:03:04.583674 10372 tablet_bootstrap.cc:492] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260: Bootstrap starting.
I20251214 03:03:04.583901 10371 tablet_bootstrap.cc:654] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:04.583878 10276 tablet_service.cc:1505] Processing CreateTablet for tablet 0442b9d36d63446ba21ad3816d7a91bd (DEFAULT_TABLE table=test-table [id=4d082de0c00e4cba9da80a3d390f559d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:04.584110 10276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0442b9d36d63446ba21ad3816d7a91bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:04.584456 10372 tablet_bootstrap.cc:654] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:04.585088 10371 tablet_bootstrap.cc:492] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b: No bootstrap required, opened a new log
I20251214 03:03:04.585232 10371 ts_tablet_manager.cc:1403] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251214 03:03:04.585641 10371 raft_consensus.cc:359] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.585737 10371 raft_consensus.cc:385] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:04.585767 10371 raft_consensus.cc:740] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152aa837d18449d891ba835a3cc0420b, State: Initialized, Role: FOLLOWER
I20251214 03:03:04.585888 10371 consensus_queue.cc:260] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.586120 10371 ts_tablet_manager.cc:1434] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251214 03:03:04.586236 10370 tablet_bootstrap.cc:492] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0: Bootstrap starting.
I20251214 03:03:04.587112 10370 tablet_bootstrap.cc:654] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:04.587744 10372 tablet_bootstrap.cc:492] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260: No bootstrap required, opened a new log
I20251214 03:03:04.587836 10372 ts_tablet_manager.cc:1403] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20251214 03:03:04.588162 10370 tablet_bootstrap.cc:492] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0: No bootstrap required, opened a new log
I20251214 03:03:04.588243 10370 ts_tablet_manager.cc:1403] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251214 03:03:04.588267 10372 raft_consensus.cc:359] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.588368 10372 raft_consensus.cc:385] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:04.588397 10372 raft_consensus.cc:740] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae10355c88b4e40a02e6016c9b2d260, State: Initialized, Role: FOLLOWER
I20251214 03:03:04.588513 10372 consensus_queue.cc:260] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.588644 10370 raft_consensus.cc:359] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.588742 10370 raft_consensus.cc:385] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:04.588770 10370 raft_consensus.cc:740] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a9b80de17e4062846ac8b3f874d6b0, State: Initialized, Role: FOLLOWER
I20251214 03:03:04.588783 10372 ts_tablet_manager.cc:1434] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251214 03:03:04.588879 10370 consensus_queue.cc:260] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.589124 10370 ts_tablet_manager.cc:1434] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 03:03:04.684391 10391 raft_consensus.cc:493] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:03:04.684557 10391 raft_consensus.cc:515] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.685021 10391 leader_election.cc:290] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519), a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545)
I20251214 03:03:04.685566 10154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0442b9d36d63446ba21ad3816d7a91bd" candidate_uuid: "152aa837d18449d891ba835a3cc0420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae10355c88b4e40a02e6016c9b2d260" is_pre_election: true
I20251214 03:03:04.685597 10296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0442b9d36d63446ba21ad3816d7a91bd" candidate_uuid: "152aa837d18449d891ba835a3cc0420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" is_pre_election: true
I20251214 03:03:04.685709 10154 raft_consensus.cc:2468] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152aa837d18449d891ba835a3cc0420b in term 0.
I20251214 03:03:04.685784 10296 raft_consensus.cc:2468] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152aa837d18449d891ba835a3cc0420b in term 0.
I20251214 03:03:04.686002 9943 leader_election.cc:304] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152aa837d18449d891ba835a3cc0420b, bae10355c88b4e40a02e6016c9b2d260; no voters:
I20251214 03:03:04.686183 10391 raft_consensus.cc:2804] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:03:04.686235 10391 raft_consensus.cc:493] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:03:04.686291 10391 raft_consensus.cc:3060] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:04.687357 10391 raft_consensus.cc:515] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.687737 10391 leader_election.cc:290] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [CANDIDATE]: Term 1 election: Requested vote from peers bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519), a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545)
I20251214 03:03:04.688143 10296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0442b9d36d63446ba21ad3816d7a91bd" candidate_uuid: "152aa837d18449d891ba835a3cc0420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a9b80de17e4062846ac8b3f874d6b0"
I20251214 03:03:04.688164 10154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0442b9d36d63446ba21ad3816d7a91bd" candidate_uuid: "152aa837d18449d891ba835a3cc0420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae10355c88b4e40a02e6016c9b2d260"
I20251214 03:03:04.688282 10296 raft_consensus.cc:3060] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:04.688303 10154 raft_consensus.cc:3060] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:04.689455 10296 raft_consensus.cc:2468] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152aa837d18449d891ba835a3cc0420b in term 1.
I20251214 03:03:04.689560 10154 raft_consensus.cc:2468] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152aa837d18449d891ba835a3cc0420b in term 1.
I20251214 03:03:04.689848 9945 leader_election.cc:304] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152aa837d18449d891ba835a3cc0420b, a9a9b80de17e4062846ac8b3f874d6b0; no voters:
I20251214 03:03:04.690099 10391 raft_consensus.cc:2804] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:04.690625 10391 raft_consensus.cc:697] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 1 LEADER]: Becoming Leader. State: Replica: 152aa837d18449d891ba835a3cc0420b, State: Running, Role: LEADER
I20251214 03:03:04.690840 10391 consensus_queue.cc:237] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } }
I20251214 03:03:04.692724 9847 catalog_manager.cc:5654] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b reported cstate change: term changed from 0 to 1, leader changed from <none> to 152aa837d18449d891ba835a3cc0420b (127.1.157.193). New cstate: current_term: 1 leader_uuid: "152aa837d18449d891ba835a3cc0420b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152aa837d18449d891ba835a3cc0420b" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 } health_report { overall_health: UNKNOWN } } }
I20251214 03:03:04.724479 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.721807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.1.157.193:34591)
1214 03:03:04.722021 (+ 214us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.722025 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.722093 (+ 68us) server_negotiation.cc:408] Connection header received
1214 03:03:04.722252 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.722255 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.722300 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.722367 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:04.722707 (+ 340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.723243 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.723862 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.724031 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.724071 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.724134 (+ 63us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.724193 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.724292 (+ 99us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.724337 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20251214 03:03:04.733537 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.730882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:44907 (local address 127.1.157.195:41545)
1214 03:03:04.731012 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.731016 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.731115 (+ 99us) server_negotiation.cc:408] Connection header received
1214 03:03:04.731228 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.731232 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.731287 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.731363 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:04.731710 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.732167 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.732846 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.732986 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.733030 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.733097 (+ 67us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.733158 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.733385 (+ 227us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.733433 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251214 03:03:04.733544 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.730780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41545 (local address 127.1.157.193:44907)
1214 03:03:04.731042 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:04.731055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:04.731144 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:04.731376 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:04.731379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:04.731385 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:03:04.731581 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.731586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.732318 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.732321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:04.732738 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.732745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.732833 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.732881 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.733254 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.733260 (+ 6us) client_negotiation.cc:769] Sending connection context
1214 03:03:04.733334 (+ 74us) client_negotiation.cc:240] Negotiation successful
1214 03:03:04.733417 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20251214 03:03:04.739106 10296 raft_consensus.cc:1275] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Refusing update from remote peer 152aa837d18449d891ba835a3cc0420b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:04.739182 10154 raft_consensus.cc:1275] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Refusing update from remote peer 152aa837d18449d891ba835a3cc0420b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:04.739468 10391 consensus_queue.cc:1048] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a9b80de17e4062846ac8b3f874d6b0" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:04.739614 10457 consensus_queue.cc:1048] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bae10355c88b4e40a02e6016c9b2d260" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:04.754076 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.751488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:46939 (local address 127.1.157.193:34591)
1214 03:03:04.751610 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:04.751614 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:04.751763 (+ 149us) server_negotiation.cc:408] Connection header received
1214 03:03:04.751875 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:04.751878 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:04.751935 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:04.752008 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:04.752347 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.752826 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.753454 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.753622 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.753662 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.753735 (+ 73us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.753806 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:04.753919 (+ 113us) server_negotiation.cc:300] Negotiation successful
1214 03:03:04.753972 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251214 03:03:04.754103 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.751423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:34591 (local address 127.1.157.195:46939)
1214 03:03:04.751681 (+ 258us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:04.751693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:04.751793 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:04.752027 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:04.752029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:04.752038 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1214 03:03:04.752247 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.752253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.752924 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:04.752928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:04.753353 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:04.753360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:04.753457 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:04.753486 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1214 03:03:04.753836 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1214 03:03:04.753842 (+ 6us) client_negotiation.cc:769] Sending connection context
1214 03:03:04.753894 (+ 52us) client_negotiation.cc:240] Negotiation successful
1214 03:03:04.753965 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20251214 03:03:04.773100 9847 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:04.773298 9847 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:04.775193 9847 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 9fef168f76e24c65bb15a9a738c1f3f3: Sending DeleteTablet for 3 replicas of tablet 0442b9d36d63446ba21ad3816d7a91bd
I20251214 03:03:04.775697 9992 tablet_service.cc:1552] Processing DeleteTablet for tablet 0442b9d36d63446ba21ad3816d7a91bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:04 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:55440
I20251214 03:03:04.775830 10134 tablet_service.cc:1552] Processing DeleteTablet for tablet 0442b9d36d63446ba21ad3816d7a91bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:04 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:38228
I20251214 03:03:04.775872 10396 tablet_replica.cc:333] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b: stopping tablet replica
I20251214 03:03:04.775857 10276 tablet_service.cc:1552] Processing DeleteTablet for tablet 0442b9d36d63446ba21ad3816d7a91bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:04 UTC) from {username='oryx', principal='oryx/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:50856
I20251214 03:03:04.775986 10396 raft_consensus.cc:2243] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 1 LEADER]: Raft consensus shutting down.
I20251214 03:03:04.776013 10394 tablet_replica.cc:333] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0: stopping tablet replica
I20251214 03:03:04.776085 10395 tablet_replica.cc:333] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260: stopping tablet replica
I20251214 03:03:04.776104 10394 raft_consensus.cc:2243] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:04.776171 10396 raft_consensus.cc:2272] T 0442b9d36d63446ba21ad3816d7a91bd P 152aa837d18449d891ba835a3cc0420b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:04.776192 10395 raft_consensus.cc:2243] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:04.776216 10394 raft_consensus.cc:2272] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:04.776311 10395 raft_consensus.cc:2272] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:04.776639 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 9927
I20251214 03:03:04.776927 10394 ts_tablet_manager.cc:1916] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:04.777002 10395 ts_tablet_manager.cc:1916] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:03:04.779531 10394 ts_tablet_manager.cc:1929] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:03:04.779604 10394 log.cc:1199] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/0442b9d36d63446ba21ad3816d7a91bd
I20251214 03:03:04.779781 10395 ts_tablet_manager.cc:1929] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251214 03:03:04.779836 10395 log.cc:1199] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765681312865542-1655-0/minicluster-data/ts-1/wal/wals/0442b9d36d63446ba21ad3816d7a91bd
I20251214 03:03:04.779851 10394 ts_tablet_manager.cc:1950] T 0442b9d36d63446ba21ad3816d7a91bd P a9a9b80de17e4062846ac8b3f874d6b0: Deleting consensus metadata
I20251214 03:03:04.780004 10395 ts_tablet_manager.cc:1950] T 0442b9d36d63446ba21ad3816d7a91bd P bae10355c88b4e40a02e6016c9b2d260: Deleting consensus metadata
I20251214 03:03:04.780691 9839 catalog_manager.cc:4985] TS a9a9b80de17e4062846ac8b3f874d6b0 (127.1.157.195:41545): tablet 0442b9d36d63446ba21ad3816d7a91bd (table test-table [id=4d082de0c00e4cba9da80a3d390f559d]) successfully deleted
I20251214 03:03:04.780884 9837 catalog_manager.cc:4985] TS bae10355c88b4e40a02e6016c9b2d260 (127.1.157.194:39519): tablet 0442b9d36d63446ba21ad3816d7a91bd (table test-table [id=4d082de0c00e4cba9da80a3d390f559d]) successfully deleted
W20251214 03:03:04.785622 9837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.157.193:34591 (error 108)
I20251214 03:03:04.785756 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 10069
I20251214 03:03:04.787588 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:04.786855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:34591 (local address 127.0.0.1:55464)
1214 03:03:04.787410 (+ 555us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:04.787476 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.193:34591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":60,"threads_started":1}
W20251214 03:03:04.787739 9837 catalog_manager.cc:4712] TS 152aa837d18449d891ba835a3cc0420b (127.1.157.193:34591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0442b9d36d63446ba21ad3816d7a91bd: Network error: Client connection negotiation failed: client connection to 127.1.157.193:34591: connect: Connection refused (error 111)
I20251214 03:03:04.792766 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 10211
I20251214 03:03:04.799006 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 9821
2025-12-14T03:03:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4359 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:03:04 dist-test-slave-k7z7 krb5kdc[10473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:03:04 dist-test-slave-k7z7 krb5kdc[10473](info): set up 2 sockets
Dec 14 03:03:04 dist-test-slave-k7z7 krb5kdc[10473](info): commencing operation
krb5kdc: starting...
W20251214 03:03:06.634083 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.813s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:03:06 dist-test-slave-k7z7 krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:03:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:03:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:03:06.768466 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:42943
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:44035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:42943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:06.874094 10489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:06.874316 10489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:06.874390 10489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:06.877524 10489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:03:06.877575 10489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:06.877605 10489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:06.877630 10489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:03:06.877661 10489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:03:06.881686 10489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:44035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:42943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:42943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:06.882655 10489 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:06.883445 10489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:06.888481 10494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:06.888563 10495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:06.888500 10497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:06.888679 10489 server_base.cc:1047] running on GCE node
I20251214 03:03:06.889230 10489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:06.890059 10489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:06.891194 10489 hybrid_clock.cc:648] HybridClock initialized: now 1765681386891175 us; error 34 us; skew 500 ppm
W20251214 03:03:06.894090 10489 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-12-14T03:03:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2096 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:03:06 dist-test-slave-k7z7 krb5kdc[10502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:03:06 dist-test-slave-k7z7 krb5kdc[10502](info): set up 2 sockets
Dec 14 03:03:06 dist-test-slave-k7z7 krb5kdc[10502](info): commencing operation
krb5kdc: starting...
W20251214 03:03:08.738077 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.821s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:03:08 dist-test-slave-k7z7 krb5kdc[10502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:03:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:03:08.871590 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:44187
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:44187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:08.975565 10518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:08.975791 10518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:08.975865 10518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:08.979122 10518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:03:08.979188 10518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:08.979219 10518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:08.979252 10518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:03:08.979285 10518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:03:08.983340 10518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:44187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:44187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:08.984300 10518 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:08.985124 10518 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:08.990240 10526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:08.990247 10524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:08.990216 10523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:08.990588 10518 server_base.cc:1047] running on GCE node
I20251214 03:03:08.991043 10518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:08.991875 10518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:08.993037 10518 hybrid_clock.cc:648] HybridClock initialized: now 1765681388993019 us; error 35 us; skew 500 ppm
Dec 14 03:03:08 dist-test-slave-k7z7 krb5kdc[10502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681388, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:08.998903 10518 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:03:09.000206 10518 webserver.cc:492] Webserver started at http://127.1.157.254:35565/ using document root <none> and password file <none>
I20251214 03:03:09.000777 10518 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:09.000829 10518 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:09.001013 10518 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:09.002779 10518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a"
format_stamp: "Formatted at 2025-12-14 03:03:08 on dist-test-slave-k7z7"
server_key: "b692f21556c1bd80e42188cb233c9880"
server_key_iv: "a608e6e287b545fad32b3d011ef1f7d5"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.003232 10518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a"
format_stamp: "Formatted at 2025-12-14 03:03:08 on dist-test-slave-k7z7"
server_key: "b692f21556c1bd80e42188cb233c9880"
server_key_iv: "a608e6e287b545fad32b3d011ef1f7d5"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.006696 10518 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:03:09.009399 10533 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:09.010773 10518 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:03:09.010890 10518 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a"
format_stamp: "Formatted at 2025-12-14 03:03:08 on dist-test-slave-k7z7"
server_key: "b692f21556c1bd80e42188cb233c9880"
server_key_iv: "a608e6e287b545fad32b3d011ef1f7d5"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.010993 10518 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:09.034468 10518 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:09.038290 10518 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:09.038630 10518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:09.046875 10518 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:44187
I20251214 03:03:09.046875 10585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:44187 every 8 connection(s)
I20251214 03:03:09.048144 10518 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:03:09.051198 10586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:09.056864 10586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a: Bootstrap starting.
I20251214 03:03:09.057314 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 10518
I20251214 03:03:09.057427 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:03:09.057664 1655 external_mini_cluster.cc:1468] Setting key 9cb8d83f7ceb97aace0ba2e10916b2aa
I20251214 03:03:09.059727 10586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:09.060472 10586 log.cc:826] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:09.063318 10586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a: No bootstrap required, opened a new log
Dec 14 03:03:09 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:03:09.066708 10586 raft_consensus.cc:359] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } }
I20251214 03:03:09.067111 10586 raft_consensus.cc:385] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:09.067215 10586 raft_consensus.cc:740] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb8e1f7abef427bb1a83e5c1b621d5a, State: Initialized, Role: FOLLOWER
I20251214 03:03:09.067751 10586 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } }
I20251214 03:03:09.067924 10586 raft_consensus.cc:399] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:03:09.067993 10586 raft_consensus.cc:493] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:03:09.068084 10586 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:09.069687 10586 raft_consensus.cc:515] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } }
I20251214 03:03:09.070091 10586 leader_election.cc:304] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb8e1f7abef427bb1a83e5c1b621d5a; no voters:
I20251214 03:03:09.070520 10586 leader_election.cc:290] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:03:09.070677 10591 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:09.070948 10591 raft_consensus.cc:697] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 1 LEADER]: Becoming Leader. State: Replica: cdb8e1f7abef427bb1a83e5c1b621d5a, State: Running, Role: LEADER
I20251214 03:03:09.071368 10591 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } }
I20251214 03:03:09.071806 10586 sys_catalog.cc:565] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:03:09.072386 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:09.059145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.1.157.254:44187)
1214 03:03:09.059589 (+ 444us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:09.059599 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:09.059635 (+ 36us) server_negotiation.cc:408] Connection header received
1214 03:03:09.060183 (+ 548us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:09.060201 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:09.060508 (+ 307us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:09.060828 (+ 320us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:09.061793 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.062585 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:09.063730 (+ 1145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.063982 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:09.066553 (+ 2571us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:09.066577 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:09.066590 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:09.066630 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:09.068981 (+ 2351us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:09.069540 (+ 559us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:09.069546 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:09.069553 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:09.069660 (+ 107us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:09.070137 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:09.070141 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:09.070143 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:09.070659 (+ 516us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:09.070879 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:09.071268 (+ 389us) server_negotiation.cc:300] Negotiation successful
1214 03:03:09.071559 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":158,"threads_started":1}
I20251214 03:03:09.072508 10591 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb8e1f7abef427bb1a83e5c1b621d5a. Latest consensus state: current_term: 1 leader_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } } }
I20251214 03:03:09.072626 10591 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:09.072824 10592 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } } }
I20251214 03:03:09.072904 10592 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:09.073402 10595 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:03:09.076007 10595 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:03:09.081492 10595 catalog_manager.cc:1357] Generated new cluster ID: e7de14cc0fd04e089d9e43477a72e9da
I20251214 03:03:09.081575 10595 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:03:09.095484 10595 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:03:09.096442 10595 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:03:09.102681 10595 catalog_manager.cc:6027] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a: Generated new TSK 0
I20251214 03:03:09.103410 10595 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:03:09.172365 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:44187
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:09.299046 10614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:09.299288 10614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:09.299345 10614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:09.302603 10614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:09.302670 10614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:09.302748 10614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:03:09.306937 10614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:09.307927 10614 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:09.308750 10614 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:09.315150 10620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:09.315201 10622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:09.315320 10619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:09.315512 10614 server_base.cc:1047] running on GCE node
I20251214 03:03:09.315833 10614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:09.316365 10614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:09.317509 10614 hybrid_clock.cc:648] HybridClock initialized: now 1765681389317500 us; error 31 us; skew 500 ppm
Dec 14 03:03:09 dist-test-slave-k7z7 krb5kdc[10502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:09.320005 10614 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:03:09.321031 10614 webserver.cc:492] Webserver started at http://127.1.157.193:42185/ using document root <none> and password file <none>
I20251214 03:03:09.321587 10614 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:09.321637 10614 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:09.321815 10614 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:09.323493 10614 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "69c4de0f330f4ac9a826494600b574eb"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "5ab05815c858b187d634d54b9b4ece40"
server_key_iv: "8bfd8321be9875f86ee4f3403bb64c6c"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.323944 10614 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "69c4de0f330f4ac9a826494600b574eb"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "5ab05815c858b187d634d54b9b4ece40"
server_key_iv: "8bfd8321be9875f86ee4f3403bb64c6c"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.327270 10614 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:03:09.329329 10629 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:09.330257 10614 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:09.330344 10614 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "69c4de0f330f4ac9a826494600b574eb"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "5ab05815c858b187d634d54b9b4ece40"
server_key_iv: "8bfd8321be9875f86ee4f3403bb64c6c"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.330428 10614 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:09.347782 10614 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:09.353255 10614 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:09.353412 10614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:09.353914 10614 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:09.354743 10614 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:09.354789 10614 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:09.354825 10614 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:09.354892 10614 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:09.363137 10614 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:45949
I20251214 03:03:09.363168 10742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:45949 every 8 connection(s)
I20251214 03:03:09.364061 10614 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:03:09.369107 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 10614
I20251214 03:03:09.369251 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:03:09.369509 1655 external_mini_cluster.cc:1468] Setting key 709a723fe2729badfc1eff61b164e46a
Dec 14 03:03:09 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:03:09.375718 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:09.365742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:45139 (local address 127.1.157.254:44187)
1214 03:03:09.365874 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:09.365877 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:09.366493 (+ 616us) server_negotiation.cc:408] Connection header received
1214 03:03:09.367245 (+ 752us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:09.367248 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:09.367294 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:09.367364 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:09.368862 (+ 1498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.369619 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:09.370277 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.370452 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:09.372498 (+ 2046us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:09.372524 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:09.372528 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:09.372555 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:09.373914 (+ 1359us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:09.374434 (+ 520us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:09.374437 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:09.374438 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:09.374471 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:09.374762 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:09.374764 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:09.374765 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:09.374892 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:09.374959 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:09.375499 (+ 540us) server_negotiation.cc:300] Negotiation successful
1214 03:03:09.375600 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251214 03:03:09.376497 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:09.365976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:45139)
1214 03:03:09.366362 (+ 386us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:09.366394 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:09.367083 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:09.367489 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:09.367496 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:09.367858 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:09.368734 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:09.368747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.369723 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:09.369726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:09.370182 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:09.370191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.370352 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:09.370938 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:09.370961 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:09.372357 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:09.374025 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:09.374030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:09.374041 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:09.374348 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:09.374560 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:09.374563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:09.374565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:09.374679 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:09.374981 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:09.374988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:09.375268 (+ 280us) client_negotiation.cc:769] Sending connection context
1214 03:03:09.375487 (+ 219us) client_negotiation.cc:240] Negotiation successful
1214 03:03:09.375742 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20251214 03:03:09.377571 10743 heartbeater.cc:344] Connected to a master server at 127.1.157.254:44187
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:03:09.377789 10743 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:09.378240 10743 heartbeater.cc:507] Master 127.1.157.254:44187 requested a full tablet report, sending...
I20251214 03:03:09.379537 10550 ts_manager.cc:194] Registered new tserver with Master: 69c4de0f330f4ac9a826494600b574eb (127.1.157.193:45949)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
I20251214 03:03:09.380640 10550 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:45139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:03:09.410473 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:44187
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:09.520887 10750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:09.521132 10750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:09.521220 10750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:09.524642 10750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:09.524718 10750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:09.524801 10750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:03:09.529292 10750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:09.530383 10750 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:09.531230 10750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:09.537704 10756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:09.537825 10755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:09.537906 10750 server_base.cc:1047] running on GCE node
W20251214 03:03:09.537705 10758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:09.538323 10750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:09.538822 10750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:09.539981 10750 hybrid_clock.cc:648] HybridClock initialized: now 1765681389539929 us; error 63 us; skew 500 ppm
Dec 14 03:03:09 dist-test-slave-k7z7 krb5kdc[10502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:09.542563 10750 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:03:09.543617 10750 webserver.cc:492] Webserver started at http://127.1.157.194:34049/ using document root <none> and password file <none>
I20251214 03:03:09.544138 10750 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:09.544202 10750 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:09.544386 10750 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:09.546154 10750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "48058bd492c84a938d5ba0a2455479e0"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "b4d3600dd37a5f976cf5def029e77895"
server_key_iv: "f4a7d7886e7b223b41ad29c610cb4505"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.546628 10750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "48058bd492c84a938d5ba0a2455479e0"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "b4d3600dd37a5f976cf5def029e77895"
server_key_iv: "f4a7d7886e7b223b41ad29c610cb4505"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.549944 10750 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:03:09.552076 10765 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:09.553009 10750 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:09.553108 10750 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "48058bd492c84a938d5ba0a2455479e0"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "b4d3600dd37a5f976cf5def029e77895"
server_key_iv: "f4a7d7886e7b223b41ad29c610cb4505"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.553241 10750 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:09.573293 10750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:09.575964 10750 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:09.576114 10750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:09.576596 10750 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:09.577432 10750 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:09.577481 10750 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:09.577529 10750 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:09.577549 10750 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:09.585737 10750 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:34147
I20251214 03:03:09.585886 10878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:34147 every 8 connection(s)
I20251214 03:03:09.586673 10750 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
Dec 14 03:03:09 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:03:09.596144 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 10750
I20251214 03:03:09.596249 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:03:09.596472 1655 external_mini_cluster.cc:1468] Setting key 9ef94a27f95075bd46dff4da03cd52bf
I20251214 03:03:09.599413 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:09.588316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:55679 (local address 127.1.157.254:44187)
1214 03:03:09.588458 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:09.588463 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:09.589144 (+ 681us) server_negotiation.cc:408] Connection header received
1214 03:03:09.589982 (+ 838us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:09.589985 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:09.590032 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:09.590104 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:09.591734 (+ 1630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.592243 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:09.592912 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.593067 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:09.595264 (+ 2197us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:09.595284 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:09.595286 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:09.595311 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:09.596690 (+ 1379us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:09.597255 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:09.597258 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:09.597260 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:09.597309 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:09.597652 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:09.597655 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:09.597657 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:09.597803 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:09.597890 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:09.598961 (+ 1071us) server_negotiation.cc:300] Negotiation successful
1214 03:03:09.599240 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251214 03:03:09.600101 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:09.588606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:55679)
1214 03:03:09.589015 (+ 409us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:09.589049 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:09.589810 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:09.590262 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:09.590269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:09.590640 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:09.591567 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:09.591581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.592366 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:09.592370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:09.592783 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:09.592790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.592970 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:09.593625 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:09.593652 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:09.595097 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:09.596807 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:09.596814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:09.596826 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:09.597116 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:09.597437 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:09.597441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:09.597442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:09.597546 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:09.597907 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:09.597915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:09.598159 (+ 244us) client_negotiation.cc:769] Sending connection context
1214 03:03:09.598834 (+ 675us) client_negotiation.cc:240] Negotiation successful
1214 03:03:09.599367 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20251214 03:03:09.601326 10879 heartbeater.cc:344] Connected to a master server at 127.1.157.254:44187
I20251214 03:03:09.601586 10879 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:09.602185 10879 heartbeater.cc:507] Master 127.1.157.254:44187 requested a full tablet report, sending...
I20251214 03:03:09.603323 10550 ts_manager.cc:194] Registered new tserver with Master: 48058bd492c84a938d5ba0a2455479e0 (127.1.157.194:34147)
I20251214 03:03:09.603906 10550 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:55679
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:03:09.644793 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:44187
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:09.752103 10886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:09.752322 10886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:09.752383 10886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:09.755667 10886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:09.755731 10886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:09.755824 10886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:03:09.760020 10886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:09.761016 10886 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:09.761852 10886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:09.768119 10891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:09.768126 10894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:09.768148 10892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:09.768447 10886 server_base.cc:1047] running on GCE node
I20251214 03:03:09.768786 10886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:09.769337 10886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:09.770505 10886 hybrid_clock.cc:648] HybridClock initialized: now 1765681389770491 us; error 29 us; skew 500 ppm
Dec 14 03:03:09 dist-test-slave-k7z7 krb5kdc[10502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:09.773108 10886 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:03:09.774246 10886 webserver.cc:492] Webserver started at http://127.1.157.195:42921/ using document root <none> and password file <none>
I20251214 03:03:09.774780 10886 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:09.774832 10886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:09.774998 10886 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:09.776639 10886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "25cce39f92e8497fb8c4fe7eb536a052"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "d73a71f4f51461ddba547e8d43b2e0df"
server_key_iv: "df577db2178296329d1d4b887a66779d"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.777078 10886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "25cce39f92e8497fb8c4fe7eb536a052"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "d73a71f4f51461ddba547e8d43b2e0df"
server_key_iv: "df577db2178296329d1d4b887a66779d"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.780340 10886 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251214 03:03:09.782419 10901 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:09.783356 10886 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:03:09.783452 10886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "25cce39f92e8497fb8c4fe7eb536a052"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "d73a71f4f51461ddba547e8d43b2e0df"
server_key_iv: "df577db2178296329d1d4b887a66779d"
server_key_version: "encryptionkey@0"
I20251214 03:03:09.783535 10886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:09.813881 10886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:09.816553 10886 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:09.816717 10886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:09.817272 10886 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:09.818100 10886 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:09.818149 10886 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:09.818212 10886 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:09.818236 10886 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:09.826503 10886 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:43811
I20251214 03:03:09.826529 11014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:43811 every 8 connection(s)
I20251214 03:03:09.827353 10886 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:03:09.830516 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 10886
I20251214 03:03:09.830621 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:03:09.830865 1655 external_mini_cluster.cc:1468] Setting key fd105bdedf3e4bf7907e54a76998caf5
Dec 14 03:03:09 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:03:09.838464 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:09.828852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:39833 (local address 127.1.157.254:44187)
1214 03:03:09.828966 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:09.828970 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:09.829639 (+ 669us) server_negotiation.cc:408] Connection header received
1214 03:03:09.830451 (+ 812us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:09.830454 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:09.830524 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:09.830599 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:09.832466 (+ 1867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.833009 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:09.833704 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.833874 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:09.835841 (+ 1967us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:09.835855 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:09.835857 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:09.835880 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:09.837011 (+ 1131us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:09.837453 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:09.837456 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:09.837457 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:09.837495 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:09.837689 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:09.837691 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:09.837692 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:09.837811 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:09.837875 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:09.838264 (+ 389us) server_negotiation.cc:300] Negotiation successful
1214 03:03:09.838351 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251214 03:03:09.838996 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:09.829106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:39833)
1214 03:03:09.829523 (+ 417us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:09.829556 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:09.830254 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:09.830790 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:09.830800 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:09.831265 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:09.832339 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:09.832353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.833116 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:09.833119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:09.833602 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:09.833610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:09.833799 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:09.834355 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:09.834376 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:09.835710 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:09.837100 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:09.837104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:09.837112 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:09.837371 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:09.837562 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:09.837564 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:09.837566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:09.837635 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:09.837872 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:09.837877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:09.838062 (+ 185us) client_negotiation.cc:769] Sending connection context
1214 03:03:09.838205 (+ 143us) client_negotiation.cc:240] Negotiation successful
1214 03:03:09.838390 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":103,"threads_started":1}
I20251214 03:03:09.839907 11015 heartbeater.cc:344] Connected to a master server at 127.1.157.254:44187
I20251214 03:03:09.840130 11015 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:09.840608 11015 heartbeater.cc:507] Master 127.1.157.254:44187 requested a full tablet report, sending...
I20251214 03:03:09.841533 10550 ts_manager.cc:194] Registered new tserver with Master: 25cce39f92e8497fb8c4fe7eb536a052 (127.1.157.195:43811)
I20251214 03:03:09.842053 10550 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:39833
I20251214 03:03:09.844445 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.193.keytab.
Entry for principal oryx/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.193.keytab.
WARNING: no policy specified for oryx/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.194.keytab.
Entry for principal oryx/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.194.keytab.
WARNING: no policy specified for oryx/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.195.keytab.
Entry for principal oryx/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.195.keytab.
I20251214 03:03:09.911922 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 10614
I20251214 03:03:09.916709 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 10750
I20251214 03:03:09.921034 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 10886
I20251214 03:03:09.925828 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 10518
I20251214 03:03:09.931135 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:44187
--webserver_interface=127.1.157.254
--webserver_port=35565
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:44187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:10.036510 11024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:10.036733 11024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:10.036795 11024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:10.040014 11024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:03:10.040071 11024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:10.040093 11024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:10.040112 11024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:03:10.040143 11024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:03:10.044355 11024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:44187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:44187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=35565
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:10.045464 11024 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:10.046320 11024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:10.051635 11029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:10.051643 11032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:10.051643 11030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:10.051810 11024 server_base.cc:1047] running on GCE node
I20251214 03:03:10.052294 11024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:10.053128 11024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:10.054292 11024 hybrid_clock.cc:648] HybridClock initialized: now 1765681390054276 us; error 32 us; skew 500 ppm
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681390, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:10.056725 11024 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:03:10.057705 11024 webserver.cc:492] Webserver started at http://127.1.157.254:35565/ using document root <none> and password file <none>
I20251214 03:03:10.058192 11024 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:10.058254 11024 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:10.061445 11024 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251214 03:03:10.063184 11039 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:10.064131 11024 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:10.064235 11024 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a"
format_stamp: "Formatted at 2025-12-14 03:03:08 on dist-test-slave-k7z7"
server_key: "b692f21556c1bd80e42188cb233c9880"
server_key_iv: "a608e6e287b545fad32b3d011ef1f7d5"
server_key_version: "encryptionkey@0"
I20251214 03:03:10.064625 11024 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:10.090111 11024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:10.092872 11024 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:10.093029 11024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:10.099864 11091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:44187 every 8 connection(s)
I20251214 03:03:10.099864 11024 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:44187
I20251214 03:03:10.100896 11024 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:03:10.105381 11092 sys_catalog.cc:263] Verifying existing consensus state
I20251214 03:03:10.106530 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 11024
I20251214 03:03:10.106907 11092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a: Bootstrap starting.
I20251214 03:03:10.106947 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:45949
--local_ip_for_outbound_sockets=127.1.157.193
--tserver_master_addrs=127.1.157.254:44187
--webserver_port=42185
--webserver_interface=127.1.157.193
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.157.193
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251214 03:03:10.125731 11092 log.cc:826] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:10.133594 11092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 03:03:10.133988 11092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a: Bootstrap complete.
I20251214 03:03:10.138679 11092 raft_consensus.cc:359] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } }
I20251214 03:03:10.139122 11092 raft_consensus.cc:740] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb8e1f7abef427bb1a83e5c1b621d5a, State: Initialized, Role: FOLLOWER
I20251214 03:03:10.139707 11092 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } }
I20251214 03:03:10.139856 11092 raft_consensus.cc:399] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:03:10.139920 11092 raft_consensus.cc:493] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:03:10.140010 11092 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 1 FOLLOWER]: Advancing to term 2
I20251214 03:03:10.142827 11092 raft_consensus.cc:515] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } }
I20251214 03:03:10.143252 11092 leader_election.cc:304] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb8e1f7abef427bb1a83e5c1b621d5a; no voters:
I20251214 03:03:10.143709 11092 leader_election.cc:290] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [CANDIDATE]: Term 2 election: Requested vote from peers
I20251214 03:03:10.143838 11096 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 2 FOLLOWER]: Leader election won for term 2
I20251214 03:03:10.144176 11096 raft_consensus.cc:697] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [term 2 LEADER]: Becoming Leader. State: Replica: cdb8e1f7abef427bb1a83e5c1b621d5a, State: Running, Role: LEADER
I20251214 03:03:10.144517 11096 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } }
I20251214 03:03:10.144950 11092 sys_catalog.cc:565] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:03:10.145457 11098 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb8e1f7abef427bb1a83e5c1b621d5a. Latest consensus state: current_term: 2 leader_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } } }
I20251214 03:03:10.145569 11098 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:10.145854 11097 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb8e1f7abef427bb1a83e5c1b621d5a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 44187 } } }
I20251214 03:03:10.145939 11097 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:10.146193 11100 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:03:10.149977 11100 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:03:10.150647 11100 catalog_manager.cc:1269] Loaded cluster ID: e7de14cc0fd04e089d9e43477a72e9da
I20251214 03:03:10.150704 11100 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:03:10.153046 11100 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:03:10.153770 11100 catalog_manager.cc:6038] T 00000000000000000000000000000000 P cdb8e1f7abef427bb1a83e5c1b621d5a: Loaded TSK: 0
I20251214 03:03:10.154989 11100 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251214 03:03:10.231415 11094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:10.231637 11094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:10.231701 11094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:10.234997 11094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:10.235059 11094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:10.235155 11094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:03:10.239348 11094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.193.keytab
--principal=oryx/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:45949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=42185
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:10.240325 11094 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:10.241124 11094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:10.247895 11119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:10.248049 11094 server_base.cc:1047] running on GCE node
W20251214 03:03:10.247896 11118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:10.248045 11121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:10.248589 11094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:10.249135 11094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:10.250320 11094 hybrid_clock.cc:648] HybridClock initialized: now 1765681390250297 us; error 44 us; skew 500 ppm
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681390, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:10.252888 11094 init.cc:377] Logged in from keytab as oryx/127.1.157.193@KRBTEST.COM (short username oryx)
I20251214 03:03:10.254123 11094 webserver.cc:492] Webserver started at http://127.1.157.193:42185/ using document root <none> and password file <none>
I20251214 03:03:10.254676 11094 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:10.254740 11094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:10.258312 11094 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251214 03:03:10.260133 11128 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:10.261114 11094 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:10.261240 11094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "69c4de0f330f4ac9a826494600b574eb"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "5ab05815c858b187d634d54b9b4ece40"
server_key_iv: "8bfd8321be9875f86ee4f3403bb64c6c"
server_key_version: "encryptionkey@0"
I20251214 03:03:10.261647 11094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:10.277413 11094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:10.280295 11094 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:10.280450 11094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:10.280987 11094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:10.281911 11094 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:10.281958 11094 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:10.281998 11094 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:10.282024 11094 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:10.291894 11094 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:45949
I20251214 03:03:10.291923 11241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:45949 every 8 connection(s)
I20251214 03:03:10.292842 11094 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:03:10.293306 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 11094
I20251214 03:03:10.293833 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:34147
--local_ip_for_outbound_sockets=127.1.157.194
--tserver_master_addrs=127.1.157.254:44187
--webserver_port=34049
--webserver_interface=127.1.157.194
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.157.194
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:10.306465 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:10.294796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:34533)
1214 03:03:10.295505 (+ 709us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:10.295546 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:10.296325 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:10.297642 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:10.297652 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:10.298160 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:10.299439 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.299459 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.301141 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.301146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:10.301811 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.301822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.302073 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.302788 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:10.302827 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:10.305659 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":114,"threads_started":1}
W20251214 03:03:10.306525 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:10.294728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:34533 (local address 127.1.157.254:44187)
1214 03:03:10.295245 (+ 517us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:10.295256 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:10.295650 (+ 394us) server_negotiation.cc:408] Connection header received
1214 03:03:10.296781 (+ 1131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:10.296797 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:10.297149 (+ 352us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:10.297538 (+ 389us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:10.299649 (+ 2111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.300986 (+ 1337us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.301935 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.302213 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.305865 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:34533: BlockingRecv error: recv got EOF from 127.1.157.193:34533 (error 108)
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":106,"threads_started":1}
W20251214 03:03:10.307205 11242 heartbeater.cc:646] Failed to heartbeat to 127.1.157.254:44187 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.157.254:44187: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:10.312954 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:10.307740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:38887)
1214 03:03:10.307918 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:10.307931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:10.308016 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:10.308253 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:10.308256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:10.308423 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:10.308667 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.308674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.309648 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.309652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:10.310177 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.310186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.310299 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.310811 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:10.310826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:10.312823 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20251214 03:03:10.313084 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:10.307815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:38887 (local address 127.1.157.254:44187)
1214 03:03:10.307926 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:10.307931 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:10.307984 (+ 53us) server_negotiation.cc:408] Connection header received
1214 03:03:10.308094 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:10.308098 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:10.308146 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:10.308230 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:10.308782 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.309520 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.310289 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.310838 (+ 549us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.312970 (+ 2132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:38887: BlockingRecv error: recv got EOF from 127.1.157.193:38887 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:10.319187 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:10.313787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:45301)
1214 03:03:10.313959 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:10.313973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:10.314048 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:10.314322 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:10.314325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:10.314478 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:10.314689 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.314696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.315844 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.315848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:10.316383 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.316393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.316507 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.317047 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:10.317063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:10.319054 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20251214 03:03:10.319320 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:10.313874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:45301 (local address 127.1.157.254:44187)
1214 03:03:10.313985 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:10.313989 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:10.314005 (+ 16us) server_negotiation.cc:408] Connection header received
1214 03:03:10.314161 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:10.314164 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:10.314217 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:10.314297 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:10.314801 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.315576 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.316524 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.317098 (+ 574us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.319204 (+ 2106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:45301: BlockingRecv error: recv got EOF from 127.1.157.193:45301 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20251214 03:03:10.319759 11242 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251214 03:03:10.408012 11246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:10.408274 11246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:10.408352 11246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:10.411870 11246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:10.411962 11246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:10.412070 11246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:03:10.416656 11246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.194.keytab
--principal=oryx/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:34147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=34049
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:10.417845 11246 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:10.418710 11246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:10.425213 11252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:10.425426 11246 server_base.cc:1047] running on GCE node
W20251214 03:03:10.425213 11254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:10.425990 11251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:10.426427 11246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:10.427004 11246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:10.428226 11246 hybrid_clock.cc:648] HybridClock initialized: now 1765681390428215 us; error 29 us; skew 500 ppm
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681390, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:10.430941 11246 init.cc:377] Logged in from keytab as oryx/127.1.157.194@KRBTEST.COM (short username oryx)
I20251214 03:03:10.432016 11246 webserver.cc:492] Webserver started at http://127.1.157.194:34049/ using document root <none> and password file <none>
I20251214 03:03:10.432546 11246 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:10.432603 11246 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:10.436096 11246 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:10.438246 11261 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:10.439391 11246 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251214 03:03:10.439498 11246 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "48058bd492c84a938d5ba0a2455479e0"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "b4d3600dd37a5f976cf5def029e77895"
server_key_iv: "f4a7d7886e7b223b41ad29c610cb4505"
server_key_version: "encryptionkey@0"
I20251214 03:03:10.439950 11246 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:10.471473 11246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:10.474416 11246 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:10.474598 11246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:10.475172 11246 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:10.476065 11246 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:10.476112 11246 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:10.476151 11246 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:10.476178 11246 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:10.484963 11246 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:34147
I20251214 03:03:10.485006 11374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:34147 every 8 connection(s)
I20251214 03:03:10.485991 11246 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:03:10.490664 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 11246
I20251214 03:03:10.491210 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:43811
--local_ip_for_outbound_sockets=127.1.157.195
--tserver_master_addrs=127.1.157.254:44187
--webserver_port=42921
--webserver_interface=127.1.157.195
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.157.195
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
W20251214 03:03:10.496837 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:10.487830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:59435 (local address 127.1.157.254:44187)
1214 03:03:10.487983 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:10.487988 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:10.488777 (+ 789us) server_negotiation.cc:408] Connection header received
1214 03:03:10.489667 (+ 890us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:10.489670 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:10.489725 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:10.489795 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:10.491875 (+ 2080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.492428 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.493103 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.493285 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.496696 (+ 3411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:59435: BlockingRecv error: recv got EOF from 127.1.157.194:59435 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20251214 03:03:10.497229 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:10.488148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:59435)
1214 03:03:10.488621 (+ 473us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:10.488659 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:10.489483 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:10.489945 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:10.489954 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:10.490388 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:10.491665 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.491687 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.492572 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.492575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:10.492987 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.492994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.493220 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.493905 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:10.493930 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:10.496485 (+ 2555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
W20251214 03:03:10.497761 11375 heartbeater.cc:646] Failed to heartbeat to 127.1.157.254:44187 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.157.254:44187: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:10.502688 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:10.498214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:38259)
1214 03:03:10.498340 (+ 126us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:10.498355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:10.498438 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:10.498676 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:10.498679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:10.498829 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:10.499014 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.499020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.499728 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.499731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:10.500097 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.500103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.500204 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.500788 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:10.500800 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:10.502566 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20251214 03:03:10.502794 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:10.498316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:38259 (local address 127.1.157.254:44187)
1214 03:03:10.498432 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:10.498436 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:10.498451 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:03:10.498535 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:10.498539 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:10.498584 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:10.498647 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:10.499139 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.499626 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.500197 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.500370 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.502698 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:38259: BlockingRecv error: recv got EOF from 127.1.157.194:38259 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:10.507813 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:10.503443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:58859)
1214 03:03:10.503610 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:10.503624 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:10.503686 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:10.503901 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:10.503904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:10.504024 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:10.504197 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.504202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.505130 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.505133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:10.505547 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.505555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.505652 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.506086 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:10.506097 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:10.507719 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20251214 03:03:10.507962 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:10.503544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:58859 (local address 127.1.157.254:44187)
1214 03:03:10.503666 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:10.503670 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:10.503685 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:03:10.503754 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:10.503757 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:10.503807 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:10.503893 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:10.504292 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.505019 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.505669 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.505835 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.507856 (+ 2021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:58859: BlockingRecv error: recv got EOF from 127.1.157.194:58859 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20251214 03:03:10.508030 11375 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251214 03:03:10.601784 11378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:10.602021 11378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:10.602082 11378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:10.605365 11378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:10.605435 11378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:10.605520 11378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:03:10.609807 11378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/krb5kdc/oryx_127.1.157.195.keytab
--principal=oryx/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:43811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=42921
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:10.610926 11378 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:10.611763 11378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:10.618048 11386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:10.618120 11384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:10.618053 11383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:10.618366 11378 server_base.cc:1047] running on GCE node
I20251214 03:03:10.618703 11378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:10.619215 11378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:10.620373 11378 hybrid_clock.cc:648] HybridClock initialized: now 1765681390620354 us; error 32 us; skew 500 ppm
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681390, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:10.622853 11378 init.cc:377] Logged in from keytab as oryx/127.1.157.195@KRBTEST.COM (short username oryx)
I20251214 03:03:10.623924 11378 webserver.cc:492] Webserver started at http://127.1.157.195:42921/ using document root <none> and password file <none>
I20251214 03:03:10.624471 11378 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:10.624533 11378 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:10.628131 11378 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:10.630287 11393 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:10.631392 11378 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:10.631506 11378 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "25cce39f92e8497fb8c4fe7eb536a052"
format_stamp: "Formatted at 2025-12-14 03:03:09 on dist-test-slave-k7z7"
server_key: "d73a71f4f51461ddba547e8d43b2e0df"
server_key_iv: "df577db2178296329d1d4b887a66779d"
server_key_version: "encryptionkey@0"
I20251214 03:03:10.631942 11378 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:10.644914 11378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:10.648113 11378 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:10.648321 11378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:10.648877 11378 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:10.649794 11378 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:10.649843 11378 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:10.649895 11378 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:10.649919 11378 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:10.658951 11378 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:43811
I20251214 03:03:10.658975 11506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:43811 every 8 connection(s)
I20251214 03:03:10.659916 11378 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:03:10.667622 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 11378
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
W20251214 03:03:10.669512 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:10.661589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:38869 (local address 127.1.157.254:44187)
1214 03:03:10.661718 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:10.661721 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:10.662382 (+ 661us) server_negotiation.cc:408] Connection header received
1214 03:03:10.663203 (+ 821us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:10.663207 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:10.663269 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:10.663340 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:10.664951 (+ 1611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.665550 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.666183 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.666357 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.669377 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:38869: BlockingRecv error: recv got EOF from 127.1.157.195:38869 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20251214 03:03:10.669965 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:10.661827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:38869)
1214 03:03:10.662242 (+ 415us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:10.662277 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:10.663011 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:10.663521 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:10.663529 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:10.663903 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:10.664804 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.664818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.665659 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.665662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:10.666076 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.666084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.666302 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.666875 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:10.666900 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:10.669255 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
W20251214 03:03:10.670423 11507 heartbeater.cc:646] Failed to heartbeat to 127.1.157.254:44187 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.157.254:44187: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:10.675144 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:10.670797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:55621)
1214 03:03:10.670888 (+ 91us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:10.670901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:10.670966 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:10.671170 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:10.671172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:10.671349 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:10.671578 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.671585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.672323 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.672326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:10.672691 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.672699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.672794 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.673282 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:10.673294 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:10.675019 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20251214 03:03:10.675252 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:10.670867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:55621 (local address 127.1.157.254:44187)
1214 03:03:10.670976 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:10.670979 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:10.670993 (+ 14us) server_negotiation.cc:408] Connection header received
1214 03:03:10.671023 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:10.671039 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:10.671078 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:10.671160 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:10.671673 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.672200 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.672790 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.672958 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.675140 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:55621: BlockingRecv error: recv got EOF from 127.1.157.195:55621 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20251214 03:03:10.677081 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:10.668507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.1.157.254:44187)
1214 03:03:10.668740 (+ 233us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:10.668743 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:10.669338 (+ 595us) server_negotiation.cc:408] Connection header received
1214 03:03:10.669462 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:10.669465 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:10.669519 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:10.669605 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:10.670518 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.671166 (+ 648us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.671937 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.672108 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.673082 (+ 974us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:10.673098 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:10.673110 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:10.673142 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:10.674955 (+ 1813us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:10.675510 (+ 555us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:10.675517 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:10.675523 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:10.675630 (+ 107us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:10.675953 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:10.675956 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:10.675958 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:10.676324 (+ 366us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:10.676523 (+ 199us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:10.676737 (+ 214us) server_negotiation.cc:300] Negotiation successful
1214 03:03:10.676958 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:10 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:10.680199 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:10.675674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:51857)
1214 03:03:10.675837 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:10.675850 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:10.675916 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:10.676141 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:10.676144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:10.676300 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:10.676497 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.676505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.677304 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.677307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:10.677814 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:10.677824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.677936 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.678450 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:10.678461 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:10.680104 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20251214 03:03:10.680321 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:10.675769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:51857 (local address 127.1.157.254:44187)
1214 03:03:10.675905 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:10.675909 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:10.675921 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:03:10.676003 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:10.676006 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:10.676044 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:10.676099 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:10.676604 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.677194 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:10.677932 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:10.678083 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:10.680214 (+ 2131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:51857: BlockingRecv error: recv got EOF from 127.1.157.195:51857 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20251214 03:03:10.680521 11507 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Dec 14 03:03:11 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:11 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:11.326231 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:11.320473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:53833)
1214 03:03:11.320786 (+ 313us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:11.320803 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:11.320923 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:11.321244 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:11.321248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:11.321438 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:11.321711 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:11.321720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.322515 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:11.322520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:11.322937 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:11.322945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.323042 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:11.323785 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:11.323808 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:11.326090 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
W20251214 03:03:11.326423 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:11.320594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:53833 (local address 127.1.157.254:44187)
1214 03:03:11.320860 (+ 266us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:11.320864 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:11.320881 (+ 17us) server_negotiation.cc:408] Connection header received
1214 03:03:11.321022 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:11.321025 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:11.321083 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:11.321190 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:11.321869 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.322391 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:11.323076 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.323422 (+ 346us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:11.326281 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:53833: BlockingRecv error: recv got EOF from 127.1.157.193:53833 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
Dec 14 03:03:11 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:11 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:11.513319 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:11.508644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:33581)
1214 03:03:11.508956 (+ 312us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:11.508971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:11.509087 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:11.509352 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:11.509354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:11.509561 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:11.509787 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:11.509794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.510460 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:11.510462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:11.510848 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:11.510854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.510935 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:11.511470 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:11.511483 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:11.513159 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":76,"threads_started":1}
W20251214 03:03:11.513432 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:11.508666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:33581 (local address 127.1.157.254:44187)
1214 03:03:11.508803 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:11.508807 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:11.509045 (+ 238us) server_negotiation.cc:408] Connection header received
1214 03:03:11.509186 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:11.509189 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:11.509242 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:11.509313 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:11.509887 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.510385 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:11.510925 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.511065 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:11.513318 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:33581: BlockingRecv error: recv got EOF from 127.1.157.194:33581 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Dec 14 03:03:11 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:11 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:11.686134 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:11.681140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:47395)
1214 03:03:11.681525 (+ 385us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:11.681539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:11.681653 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:11.681896 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:11.681899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:11.682085 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:11.682318 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:11.682325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.683031 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:11.683034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:11.683438 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:11.683445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.683542 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:11.684179 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:11.684195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:11.686013 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":105,"threads_started":1}
W20251214 03:03:11.686316 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:11.681278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:47395 (local address 127.1.157.254:44187)
1214 03:03:11.681458 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:11.681463 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:11.681605 (+ 142us) server_negotiation.cc:408] Connection header received
1214 03:03:11.681738 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:11.681742 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:11.681793 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:11.681860 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:11.682435 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.682928 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:11.683537 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:11.683723 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:11.686164 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:47395: BlockingRecv error: recv got EOF from 127.1.157.195:47395 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Dec 14 03:03:12 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:12 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:12.331465 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:12.327002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:43345)
1214 03:03:12.327280 (+ 278us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:12.327294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:12.327387 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:12.327607 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:12.327610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:12.327779 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:12.327974 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:12.327979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.328687 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:12.328690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:12.329075 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:12.329081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.329163 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:12.329707 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:12.329720 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:12.331348 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
W20251214 03:03:12.331679 11517 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:12.327100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:43345 (local address 127.1.157.254:44187)
1214 03:03:12.327367 (+ 267us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:12.327370 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:12.327383 (+ 13us) server_negotiation.cc:408] Connection header received
1214 03:03:12.327463 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:12.327465 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:12.327508 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:12.327586 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:12.328056 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.328600 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:12.329209 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.329348 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:12.331535 (+ 2187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:43345: BlockingRecv error: recv got EOF from 127.1.157.193:43345 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
Dec 14 03:03:12 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:12 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:12.518927 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:12.514138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:35697)
1214 03:03:12.514463 (+ 325us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:12.514480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:12.514603 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:12.514854 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:12.514857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:12.515046 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:12.515240 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:12.515246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.515984 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:12.515987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:12.516366 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:12.516373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.516473 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:12.517056 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:12.517069 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:12.518773 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
W20251214 03:03:12.519022 11517 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:12.514264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:35697 (local address 127.1.157.254:44187)
1214 03:03:12.514447 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:12.514451 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:12.514549 (+ 98us) server_negotiation.cc:408] Connection header received
1214 03:03:12.514691 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:12.514694 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:12.514751 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:12.514816 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:12.515362 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.515878 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:12.516466 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.516650 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:12.518922 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:35697: BlockingRecv error: recv got EOF from 127.1.157.194:35697 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Dec 14 03:03:12 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:12 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:12.691234 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:12.686829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:50379)
1214 03:03:12.687079 (+ 250us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:12.687095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:12.687188 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:12.687481 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:12.687484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:12.687656 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:12.687860 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:12.687866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.688547 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:12.688550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:12.688935 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:12.688942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.689035 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:12.689544 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:12.689556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:12.691120 (+ 1564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
W20251214 03:03:12.691357 11517 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:12.686998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:50379 (local address 127.1.157.254:44187)
1214 03:03:12.687124 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:12.687127 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:12.687141 (+ 14us) server_negotiation.cc:408] Connection header received
1214 03:03:12.687272 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:12.687274 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:12.687320 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:12.687393 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:12.687955 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.688452 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:12.689019 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:12.689137 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:12.691237 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:50379: BlockingRecv error: recv got EOF from 127.1.157.195:50379 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Dec 14 03:03:13 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:13 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:13.337020 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:13.332235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:49189)
1214 03:03:13.332547 (+ 312us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:13.332560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:13.332650 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:13.332905 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:13.332908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:13.333117 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:13.333355 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:13.333362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.334094 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:13.334098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:13.334475 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:13.334492 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.334580 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:13.335135 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:13.335147 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:13.336901 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":87,"threads_started":1}
W20251214 03:03:13.337212 11521 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:13.332363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:49189 (local address 127.1.157.254:44187)
1214 03:03:13.332591 (+ 228us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:13.332594 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:13.332608 (+ 14us) server_negotiation.cc:408] Connection header received
1214 03:03:13.332728 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:13.332730 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:13.332773 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:13.332847 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:13.333473 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.333986 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:13.334610 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.334808 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:13.337056 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:49189: BlockingRecv error: recv got EOF from 127.1.157.193:49189 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":58,"threads_started":1}
Dec 14 03:03:13 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:13 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:13.524856 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:13.519817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:49397)
1214 03:03:13.520195 (+ 378us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:13.520211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:13.520327 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:13.520654 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:13.520658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:13.520858 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:13.521118 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:13.521127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.522058 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:13.522062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:13.522443 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:13.522450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.522557 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:13.523041 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:13.523053 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:13.524740 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":105,"threads_started":1}
W20251214 03:03:13.525014 11521 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:13.519942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:49397 (local address 127.1.157.254:44187)
1214 03:03:13.520096 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:13.520100 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:13.520341 (+ 241us) server_negotiation.cc:408] Connection header received
1214 03:03:13.520424 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:13.520428 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:13.520504 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:13.520575 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:13.521271 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.521952 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:13.522543 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.522673 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:13.524900 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:49397: BlockingRecv error: recv got EOF from 127.1.157.194:49397 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Dec 14 03:03:13 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:13 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:13.697525 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:13.692065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:35797)
1214 03:03:13.692344 (+ 279us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:13.692357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:13.692552 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:13.692922 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:13.692925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:13.693100 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:13.693361 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:13.693369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.694103 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:13.694106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:13.694602 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:13.694614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.694813 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:13.695456 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:13.695475 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:13.697359 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
W20251214 03:03:13.697592 11521 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:13.692210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:35797 (local address 127.1.157.254:44187)
1214 03:03:13.692339 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:13.692342 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:13.692471 (+ 129us) server_negotiation.cc:408] Connection header received
1214 03:03:13.692720 (+ 249us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:13.692723 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:13.692775 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:13.692848 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:13.693486 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.693995 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:13.694827 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:13.695011 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:13.697495 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:35797: BlockingRecv error: recv got EOF from 127.1.157.195:35797 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Dec 14 03:03:14 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:14 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:14.342931 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:14.337834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:48463)
1214 03:03:14.338131 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:14.338145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:14.338252 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:14.338484 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:14.338486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:14.338665 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:14.338861 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:14.338867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.339687 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:14.339690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:14.340164 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:14.340175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.340318 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:14.340924 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:14.340938 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:14.342815 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
W20251214 03:03:14.343080 11525 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:14.337978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:48463 (local address 127.1.157.254:44187)
1214 03:03:14.338192 (+ 214us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:14.338195 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:14.338210 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:03:14.338333 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:14.338336 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:14.338383 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:14.338462 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:14.338995 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.339549 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:14.340298 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.340484 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:14.342958 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:48463: BlockingRecv error: recv got EOF from 127.1.157.193:48463 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":58,"threads_started":1}
Dec 14 03:03:14 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:14 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:14.534298 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:14.525721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:36457)
1214 03:03:14.526004 (+ 283us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:14.526018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:14.526129 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:14.526415 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:14.526419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:14.526660 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:14.526919 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:14.526928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.527707 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:14.527711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:14.528137 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:14.528145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.528237 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:14.532200 (+ 3963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:14.532217 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:14.534160 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
W20251214 03:03:14.534595 11525 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:14.525799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:36457 (local address 127.1.157.254:44187)
1214 03:03:14.525955 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:14.525960 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:14.526086 (+ 126us) server_negotiation.cc:408] Connection header received
1214 03:03:14.526215 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:14.526217 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:14.526268 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:14.526348 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:14.527044 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.527597 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:14.528231 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.528373 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:14.534361 (+ 5988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:36457: BlockingRecv error: recv got EOF from 127.1.157.194:36457 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Dec 14 03:03:14 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:14 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:14.703320 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:14.698330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:56485)
1214 03:03:14.698654 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:14.698667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:14.698767 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:14.699099 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:14.699102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:14.699310 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:14.699512 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:14.699521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.700287 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:14.700290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:14.700688 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:14.700696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.700796 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:14.701417 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:14.701432 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:14.703183 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":94,"threads_started":1}
W20251214 03:03:14.703404 11525 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:14.698578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:56485 (local address 127.1.157.254:44187)
1214 03:03:14.698705 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:14.698709 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:14.698724 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:03:14.698891 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:14.698894 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:14.698947 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:14.699026 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:14.699663 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.700162 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:14.700791 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:14.700957 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:14.703310 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:56485: BlockingRecv error: recv got EOF from 127.1.157.195:56485 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 14 03:03:15 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:15 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:15.348829 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:15.343755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:60067)
1214 03:03:15.344073 (+ 318us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:15.344088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:15.344171 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:15.344430 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:15.344433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:15.344633 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:15.344848 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:15.344855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.345722 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:15.345728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:15.346143 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:15.346151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.346284 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:15.346834 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:15.346849 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:15.348700 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
W20251214 03:03:15.348976 11529 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:15.343874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:60067 (local address 127.1.157.254:44187)
1214 03:03:15.344114 (+ 240us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:15.344118 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:15.344147 (+ 29us) server_negotiation.cc:408] Connection header received
1214 03:03:15.344256 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:15.344260 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:15.344313 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:15.344394 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:15.344982 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.345571 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:15.346300 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.346453 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:15.348852 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:60067: BlockingRecv error: recv got EOF from 127.1.157.193:60067 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":57,"threads_started":1}
Dec 14 03:03:15 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:15 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:15.540431 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:15.535203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:55819)
1214 03:03:15.535497 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:15.535511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:15.535614 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:15.535851 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:15.535853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:15.536087 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:15.536299 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:15.536306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.537143 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:15.537147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:15.537667 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:15.537678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.537798 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:15.538380 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:15.538394 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:15.540306 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
W20251214 03:03:15.540584 11529 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:15.535385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:55819 (local address 127.1.157.254:44187)
1214 03:03:15.535518 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:15.535523 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:15.535576 (+ 53us) server_negotiation.cc:408] Connection header received
1214 03:03:15.535705 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:15.535708 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:15.535755 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:15.535814 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:15.536437 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.536999 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:15.537798 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.538005 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:15.540465 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:55819: BlockingRecv error: recv got EOF from 127.1.157.194:55819 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Dec 14 03:03:15 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:15 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:15.708935 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:15.704216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:57469)
1214 03:03:15.704508 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:15.704525 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:15.704626 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:15.704956 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:15.704960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:15.705267 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:15.705478 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:15.705492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.706283 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:15.706286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:15.706668 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:15.706674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.706761 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:15.707258 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:15.707270 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:15.708817 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
W20251214 03:03:15.709040 11529 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:15.704298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:57469 (local address 127.1.157.254:44187)
1214 03:03:15.704522 (+ 224us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:15.704528 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:15.704581 (+ 53us) server_negotiation.cc:408] Connection header received
1214 03:03:15.704738 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:15.704742 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:15.704803 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:15.704899 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:15.705600 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.706190 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:15.706753 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:15.706879 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:15.708930 (+ 2051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:57469: BlockingRecv error: recv got EOF from 127.1.157.195:57469 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Dec 14 03:03:16 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:16 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:16.354597 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:16.349595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:34353)
1214 03:03:16.349889 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:16.349905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:16.350013 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:16.350258 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:16.350261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:16.350450 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:16.350645 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:16.350651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.351389 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:16.351392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:16.351858 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:16.351869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.351996 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:16.352587 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:16.352600 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:16.354447 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
W20251214 03:03:16.354698 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:16.349660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:34353 (local address 127.1.157.254:44187)
1214 03:03:16.349922 (+ 262us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:16.349927 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:16.349957 (+ 30us) server_negotiation.cc:408] Connection header received
1214 03:03:16.350099 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:16.350102 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:16.350148 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:16.350219 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:16.350760 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.351286 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:16.351992 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.352194 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:16.354587 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:34353: BlockingRecv error: recv got EOF from 127.1.157.193:34353 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
Dec 14 03:03:16 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:16 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:16.545821 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:16.541102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:42807)
1214 03:03:16.541402 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:16.541418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:16.541556 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:16.541791 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:16.541794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:16.541985 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:16.542179 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:16.542185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.542900 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:16.542904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:16.543285 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:16.543292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.543394 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:16.543973 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:16.543986 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:16.545697 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":110,"threads_started":1}
W20251214 03:03:16.545919 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:16.541255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:42807 (local address 127.1.157.254:44187)
1214 03:03:16.541405 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:16.541410 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:16.541493 (+ 83us) server_negotiation.cc:408] Connection header received
1214 03:03:16.541642 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:16.541645 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:16.541695 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:16.541757 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:16.542290 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.542798 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:16.543388 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.543573 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:16.545823 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:42807: BlockingRecv error: recv got EOF from 127.1.157.194:42807 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Dec 14 03:03:16 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:16 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:16.714649 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:16.709703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:50775)
1214 03:03:16.709996 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:16.710010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:16.710130 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:16.710366 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:16.710369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:16.710557 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:16.710746 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:16.710752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.711475 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:16.711479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:16.711858 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:16.711865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.711963 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:16.712536 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:16.712548 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:16.714507 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
W20251214 03:03:16.714746 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:16.709792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:50775 (local address 127.1.157.254:44187)
1214 03:03:16.709961 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:16.709965 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:16.710074 (+ 109us) server_negotiation.cc:408] Connection header received
1214 03:03:16.710215 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:16.710218 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:16.710268 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:16.710332 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:16.710859 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.711372 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:16.711954 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:16.712128 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:16.714654 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:50775: BlockingRecv error: recv got EOF from 127.1.157.195:50775 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Dec 14 03:03:17 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:17 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:17.361027 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:17.355400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:37025)
1214 03:03:17.355708 (+ 308us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:17.355723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:17.355827 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:17.356077 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:17.356080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:17.356254 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:17.356513 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:17.356522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.357976 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:17.357980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:17.358374 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:17.358382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.358496 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:17.359132 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:17.359146 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:17.360904 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":88,"threads_started":1}
W20251214 03:03:17.361153 11537 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:17.355469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:37025 (local address 127.1.157.254:44187)
1214 03:03:17.355710 (+ 241us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:17.355714 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:17.355789 (+ 75us) server_negotiation.cc:408] Connection header received
1214 03:03:17.355899 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:17.355901 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:17.355946 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:17.356021 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:17.356636 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.357866 (+ 1230us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:17.358484 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.358669 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:17.361043 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:37025: BlockingRecv error: recv got EOF from 127.1.157.193:37025 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
Dec 14 03:03:17 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:17 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:17.551339 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:17.546550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:42147)
1214 03:03:17.546876 (+ 326us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:17.546889 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:17.546996 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:17.547217 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:17.547219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:17.547407 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:17.547625 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:17.547632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.548355 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:17.548359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:17.548757 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:17.548765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.548861 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:17.549455 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:17.549469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:17.551219 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":83,"threads_started":1}
W20251214 03:03:17.551523 11537 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:17.546654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:42147 (local address 127.1.157.254:44187)
1214 03:03:17.546789 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:17.546793 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:17.546948 (+ 155us) server_negotiation.cc:408] Connection header received
1214 03:03:17.547068 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:17.547071 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:17.547118 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:17.547195 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:17.547725 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.548225 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:17.548852 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.548982 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:17.551404 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:42147: BlockingRecv error: recv got EOF from 127.1.157.194:42147 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Dec 14 03:03:17 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:17 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:17.720337 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:17.715626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:43665)
1214 03:03:17.715936 (+ 310us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:17.715948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:17.716061 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:17.716349 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:17.716353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:17.716558 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:17.716826 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:17.716834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.717558 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:17.717560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:17.717950 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:17.717957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.718043 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:17.718530 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:17.718542 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:17.720193 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":71,"threads_started":1}
W20251214 03:03:17.720515 11537 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:17.715718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:43665 (local address 127.1.157.254:44187)
1214 03:03:17.715857 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:17.715861 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:17.716029 (+ 168us) server_negotiation.cc:408] Connection header received
1214 03:03:17.716154 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:17.716157 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:17.716216 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:17.716306 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:17.716931 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.717467 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:17.718033 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:17.718159 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:17.720329 (+ 2170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:43665: BlockingRecv error: recv got EOF from 127.1.157.195:43665 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 14 03:03:18 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:18 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:18.366398 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:18.361776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:36455)
1214 03:03:18.362040 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:18.362052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:18.362140 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:18.362369 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:18.362372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:18.362536 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:18.362723 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:18.362729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.363447 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:18.363450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:18.363813 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:18.363818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.363896 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:18.364414 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:18.364428 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:18.366255 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
W20251214 03:03:18.366509 11541 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:18.361954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:36455 (local address 127.1.157.254:44187)
1214 03:03:18.362161 (+ 207us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:18.362165 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:18.362195 (+ 30us) server_negotiation.cc:408] Connection header received
1214 03:03:18.362232 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:18.362234 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:18.362277 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:18.362353 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:18.362828 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.363360 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:18.363892 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.364024 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:18.366389 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:36455: BlockingRecv error: recv got EOF from 127.1.157.193:36455 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":60,"threads_started":1}
Dec 14 03:03:18 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:18 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:18.556960 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:18.552179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:35231)
1214 03:03:18.552487 (+ 308us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:18.552501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:18.552617 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:18.552879 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:18.552882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:18.553073 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:18.553359 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:18.553369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.554296 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:18.554300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:18.554728 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:18.554737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.554824 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:18.555320 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:18.555331 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:18.556849 (+ 1518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
W20251214 03:03:18.557086 11541 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:18.552298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:35231 (local address 127.1.157.254:44187)
1214 03:03:18.552455 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:18.552460 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:18.552571 (+ 111us) server_negotiation.cc:408] Connection header received
1214 03:03:18.552700 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:18.552704 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:18.552762 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:18.552847 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:18.553475 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.554197 (+ 722us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:18.554830 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.554964 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:18.556974 (+ 2010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:35231: BlockingRecv error: recv got EOF from 127.1.157.194:35231 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Dec 14 03:03:18 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:18 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:18.726065 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:18.721109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:50159)
1214 03:03:18.721409 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:18.721422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:18.721525 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:18.721794 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:18.721796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:18.721971 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:18.722171 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:18.722178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.722868 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:18.722871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:18.723247 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:18.723254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.723344 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:18.723894 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:18.723908 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:18.725912 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":103,"threads_started":1}
W20251214 03:03:18.726166 11541 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:18.721222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:50159 (local address 127.1.157.254:44187)
1214 03:03:18.721357 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:18.721360 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:18.721484 (+ 124us) server_negotiation.cc:408] Connection header received
1214 03:03:18.721610 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:18.721612 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:18.721659 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:18.721736 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:18.722285 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.722775 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:18.723360 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:18.723501 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:18.726063 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:50159: BlockingRecv error: recv got EOF from 127.1.157.195:50159 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Dec 14 03:03:19 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:19 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:19.371867 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:19.367110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:53015)
1214 03:03:19.367407 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:19.367422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:19.367530 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:19.367835 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:19.367838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:19.368016 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:19.368213 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:19.368219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.369013 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:19.369015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:19.369443 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:19.369450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.369547 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:19.370059 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:19.370072 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:19.371723 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
W20251214 03:03:19.371978 11545 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:19.367227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:53015 (local address 127.1.157.254:44187)
1214 03:03:19.367445 (+ 218us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:19.367449 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:19.367465 (+ 16us) server_negotiation.cc:408] Connection header received
1214 03:03:19.367655 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:19.367658 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:19.367710 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:19.367792 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:19.368335 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.368923 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:19.369549 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.369679 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:19.371871 (+ 2192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:53015: BlockingRecv error: recv got EOF from 127.1.157.193:53015 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":55,"threads_started":1}
Dec 14 03:03:19 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:19 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:19.565598 11546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:19.557656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:51635)
1214 03:03:19.557954 (+ 298us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:19.557969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:19.558092 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:19.558373 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:19.558375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:19.558592 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:19.558787 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:19.558793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.559516 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:19.559519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:19.559894 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:19.559900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.560001 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:19.563538 (+ 3537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:19.563553 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:19.565465 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":87,"threads_started":1}
W20251214 03:03:19.565748 11545 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:19.557764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:51635 (local address 127.1.157.254:44187)
1214 03:03:19.557941 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:19.557945 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:19.558039 (+ 94us) server_negotiation.cc:408] Connection header received
1214 03:03:19.558180 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:19.558183 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:19.558267 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:19.558332 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:19.558907 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.559409 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:19.559998 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.560177 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:19.565628 (+ 5451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:51635: BlockingRecv error: recv got EOF from 127.1.157.194:51635 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Dec 14 03:03:19 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:19 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:19.731564 11547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:19.726860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:41579)
1214 03:03:19.727124 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:19.727137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:19.727270 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:19.727549 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:19.727552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:19.727741 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:19.727941 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:19.727948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.728693 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:19.728697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:19.729085 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:19.729092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.729226 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:19.729791 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:19.729804 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:19.731443 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
W20251214 03:03:19.731698 11545 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:19.726944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:41579 (local address 127.1.157.254:44187)
1214 03:03:19.727086 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:19.727090 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:19.727204 (+ 114us) server_negotiation.cc:408] Connection header received
1214 03:03:19.727346 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:19.727348 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:19.727394 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:19.727461 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:19.728064 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.728581 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:19.729234 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:19.729394 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:19.731582 (+ 2188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:41579: BlockingRecv error: recv got EOF from 127.1.157.195:41579 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Dec 14 03:03:20 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:20 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:20.377432 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:20.372670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:47023)
1214 03:03:20.372947 (+ 277us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:20.372960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:20.373050 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:20.373309 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:20.373312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:20.373479 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:20.373673 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:20.373678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.374392 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:20.374395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:20.374771 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:20.374778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.374868 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:20.375497 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:20.375514 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:20.377314 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20251214 03:03:20.377590 11549 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:20.372771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:47023 (local address 127.1.157.254:44187)
1214 03:03:20.372978 (+ 207us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:20.372982 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:20.373011 (+ 29us) server_negotiation.cc:408] Connection header received
1214 03:03:20.373127 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:20.373129 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:20.373193 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:20.373277 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:20.373761 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.374290 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:20.374874 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.375023 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:20.377457 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:47023: BlockingRecv error: recv got EOF from 127.1.157.193:47023 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":52,"threads_started":1}
Dec 14 03:03:20 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:20 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:20.571458 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:20.566393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:35425)
1214 03:03:20.566696 (+ 303us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:20.566709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:20.566809 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:20.567043 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:20.567046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:20.567219 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:20.567423 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:20.567429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.568209 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:20.568212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:20.568600 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:20.568608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.568717 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:20.569333 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:20.569351 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:20.571347 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":108,"threads_started":1}
W20251214 03:03:20.571574 11549 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:20.566493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:35425 (local address 127.1.157.254:44187)
1214 03:03:20.566656 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:20.566661 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:20.566798 (+ 137us) server_negotiation.cc:408] Connection header received
1214 03:03:20.566888 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:20.566891 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:20.566942 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:20.567011 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:20.567552 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.568055 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:20.568713 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.568894 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:20.571465 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:35425: BlockingRecv error: recv got EOF from 127.1.157.194:35425 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Dec 14 03:03:20 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:20 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:20.737533 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:20.732370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:37083)
1214 03:03:20.732653 (+ 283us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:20.732667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:20.732761 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:20.732979 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:20.732982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:20.733153 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:20.733399 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:20.733406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.734156 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:20.734159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:20.734556 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:20.734563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.734679 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:20.735382 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:20.735397 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:20.737372 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
W20251214 03:03:20.737600 11549 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:20.732557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:37083 (local address 127.1.157.254:44187)
1214 03:03:20.732685 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:20.732689 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:20.732716 (+ 27us) server_negotiation.cc:408] Connection header received
1214 03:03:20.732840 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:20.732843 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:20.732888 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:20.732955 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:20.733537 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.734044 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:20.734694 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:20.734839 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:20.737504 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:37083: BlockingRecv error: recv got EOF from 127.1.157.195:37083 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Dec 14 03:03:21 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:21 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:21.382730 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:21.378149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:45273)
1214 03:03:21.378404 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:21.378417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:21.378513 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:21.378736 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:21.378738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:21.378905 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:21.379101 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:21.379106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.379867 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:21.379870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:21.380292 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:21.380301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.380412 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:21.380917 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:21.380929 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:21.382614 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
W20251214 03:03:21.382849 11553 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:21.378242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:45273 (local address 127.1.157.254:44187)
1214 03:03:21.378478 (+ 236us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:21.378481 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:21.378493 (+ 12us) server_negotiation.cc:408] Connection header received
1214 03:03:21.378598 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:21.378601 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:21.378644 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:21.378716 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:21.379186 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.379724 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:21.380402 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.380550 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:21.382739 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:45273: BlockingRecv error: recv got EOF from 127.1.157.193:45273 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
Dec 14 03:03:21 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:21 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:21.577376 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:21.572320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:57233)
1214 03:03:21.572692 (+ 372us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:21.572705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:21.572810 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:21.573114 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:21.573118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:21.573389 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:21.573622 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:21.573630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.574348 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:21.574351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:21.574747 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:21.574754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.574859 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:21.575400 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:21.575413 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:21.577256 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":79,"threads_started":1}
W20251214 03:03:21.577507 11553 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:21.572399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:57233 (local address 127.1.157.254:44187)
1214 03:03:21.572539 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:21.572543 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:21.572778 (+ 235us) server_negotiation.cc:408] Connection header received
1214 03:03:21.572900 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:21.572904 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:21.572961 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:21.573045 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:21.573739 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.574254 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:21.574838 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.574965 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:21.577393 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:57233: BlockingRecv error: recv got EOF from 127.1.157.194:57233 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Dec 14 03:03:21 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:21 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:21.743662 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:21.738331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:60571)
1214 03:03:21.738656 (+ 325us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:21.738673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:21.738862 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:21.739108 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:21.739111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:21.739327 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:21.739527 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:21.739534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.740388 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:21.740392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:21.740792 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:21.740800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.740913 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:21.741574 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:21.741589 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:21.743535 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":90,"threads_started":1}
W20251214 03:03:21.743834 11553 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:21.738483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:60571 (local address 127.1.157.254:44187)
1214 03:03:21.738657 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:21.738662 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:21.738813 (+ 151us) server_negotiation.cc:408] Connection header received
1214 03:03:21.738952 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:21.738955 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:21.739005 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:21.739071 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:21.739642 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.740256 (+ 614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:21.740898 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:21.741072 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:21.743702 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:60571: BlockingRecv error: recv got EOF from 127.1.157.195:60571 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Dec 14 03:03:22 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:22 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:22.388067 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:22.383498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:38391)
1214 03:03:22.383795 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:22.383812 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:22.383934 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:22.384179 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:22.384182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:22.384382 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:22.384577 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:22.384584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.385329 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:22.385333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:22.385712 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:22.385718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.385814 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:22.386323 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:22.386336 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:22.387955 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
W20251214 03:03:22.388206 11557 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:22.383578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:38391 (local address 127.1.157.254:44187)
1214 03:03:22.383814 (+ 236us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:22.383818 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:22.383891 (+ 73us) server_negotiation.cc:408] Connection header received
1214 03:03:22.384021 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:22.384024 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:22.384070 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:22.384143 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:22.384688 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.385238 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:22.385810 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.385985 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:22.388085 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:38391: BlockingRecv error: recv got EOF from 127.1.157.193:38391 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
Dec 14 03:03:22 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:22 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:22.583070 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:22.578226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:34603)
1214 03:03:22.578501 (+ 275us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:22.578515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:22.578614 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:22.578884 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:22.578886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:22.579062 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:22.579268 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:22.579274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.580018 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:22.580022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:22.580426 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:22.580434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.580550 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:22.581102 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:22.581117 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:22.582928 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
W20251214 03:03:22.583165 11557 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:22.578309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:34603 (local address 127.1.157.254:44187)
1214 03:03:22.578448 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:22.578453 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:22.578575 (+ 122us) server_negotiation.cc:408] Connection header received
1214 03:03:22.578727 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:22.578731 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:22.578782 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:22.578855 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:22.579384 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.579897 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:22.580555 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.580696 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:22.583067 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:34603: BlockingRecv error: recv got EOF from 127.1.157.194:34603 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Dec 14 03:03:22 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:22 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:22.749668 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:22.744521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:51749)
1214 03:03:22.744844 (+ 323us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:22.744860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:22.744984 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:22.745259 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:22.745262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:22.745505 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:22.745700 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:22.745707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.746531 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:22.746534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:22.747012 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:22.747022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.747123 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:22.747788 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:22.747802 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:22.749522 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
W20251214 03:03:22.749747 11557 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:22.744628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:51749 (local address 127.1.157.254:44187)
1214 03:03:22.744823 (+ 195us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:22.744828 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:22.744934 (+ 106us) server_negotiation.cc:408] Connection header received
1214 03:03:22.745078 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:22.745081 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:22.745130 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:22.745263 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:22.745814 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.746416 (+ 602us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:22.747116 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:22.747319 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:22.749650 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:51749: BlockingRecv error: recv got EOF from 127.1.157.195:51749 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Dec 14 03:03:23 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:23 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:23.393131 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:23.388769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:60189)
1214 03:03:23.389031 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:23.389044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:23.389137 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:23.389393 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:23.389396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:23.389567 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:23.389759 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:23.389764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.390460 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:23.390463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:23.390834 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:23.390840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.390922 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:23.391428 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:23.391441 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:23.393020 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
W20251214 03:03:23.393294 11561 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:23.388852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:60189 (local address 127.1.157.254:44187)
1214 03:03:23.389060 (+ 208us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:23.389063 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:23.389093 (+ 30us) server_negotiation.cc:408] Connection header received
1214 03:03:23.389254 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:23.389256 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:23.389300 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:23.389373 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:23.389847 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.390371 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:23.390910 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.391033 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:23.393142 (+ 2109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:60189: BlockingRecv error: recv got EOF from 127.1.157.193:60189 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":53,"threads_started":1}
Dec 14 03:03:23 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:23 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:23.588428 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:23.583823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:51999)
1214 03:03:23.584122 (+ 299us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:23.584138 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:23.584258 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:23.584512 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:23.584515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:23.584701 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:23.584890 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:23.584896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.585685 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:23.585689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:23.586053 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:23.586060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.586157 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:23.586674 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:23.586686 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:23.588318 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
W20251214 03:03:23.588586 11561 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:23.583989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:51999 (local address 127.1.157.254:44187)
1214 03:03:23.584125 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:23.584130 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:23.584204 (+ 74us) server_negotiation.cc:408] Connection header received
1214 03:03:23.584344 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:23.584347 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:23.584394 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:23.584458 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:23.585008 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.585581 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:23.586155 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.586347 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:23.588468 (+ 2121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:51999: BlockingRecv error: recv got EOF from 127.1.157.194:51999 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 14 03:03:23 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:23 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:23.755462 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:23.750406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:54863)
1214 03:03:23.750688 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:23.750701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:23.750813 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:23.751047 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:23.751050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:23.751260 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:23.751451 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:23.751457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.752163 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:23.752166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:23.752559 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:23.752565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.752664 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:23.753280 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:23.753293 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:23.755314 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
W20251214 03:03:23.755563 11561 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:23.750491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:54863 (local address 127.1.157.254:44187)
1214 03:03:23.750628 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:23.750632 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:23.750764 (+ 132us) server_negotiation.cc:408] Connection header received
1214 03:03:23.750900 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:23.750903 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:23.750948 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:23.751011 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:23.751575 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.752063 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:23.752661 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:23.752836 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:23.755468 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:54863: BlockingRecv error: recv got EOF from 127.1.157.195:54863 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Dec 14 03:03:24 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:24 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:24.398937 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:24.393996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:53921)
1214 03:03:24.394331 (+ 335us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:24.394346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:24.394445 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:24.394663 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:24.394665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:24.394838 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:24.395054 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:24.395059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.395923 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:24.395927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:24.396408 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:24.396416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.396512 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:24.397026 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:24.397038 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:24.398822 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":97,"threads_started":1}
W20251214 03:03:24.399070 11565 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:24.394134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:53921 (local address 127.1.157.254:44187)
1214 03:03:24.394436 (+ 302us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:24.394440 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:24.394454 (+ 14us) server_negotiation.cc:408] Connection header received
1214 03:03:24.394520 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:24.394523 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:24.394567 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:24.394638 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:24.395160 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.395817 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:24.396511 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.396656 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:24.398952 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:53921: BlockingRecv error: recv got EOF from 127.1.157.193:53921 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":66,"threads_started":1}
Dec 14 03:03:24 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:24 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:24.597992 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:24.589098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:58917)
1214 03:03:24.589402 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:24.589416 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:24.589514 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:24.589709 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:24.589711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:24.589879 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:24.590072 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:24.590078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.590740 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:24.590742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:24.591256 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:24.591263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.591368 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:24.596106 (+ 4738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:24.596120 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:24.597847 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":118,"threads_started":1}
W20251214 03:03:24.598078 11565 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:24.589210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:58917 (local address 127.1.157.254:44187)
1214 03:03:24.589337 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:24.589342 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:24.589476 (+ 134us) server_negotiation.cc:408] Connection header received
1214 03:03:24.589577 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:24.589580 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:24.589625 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:24.589687 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:24.590158 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.590657 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:24.591381 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.591600 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:24.597984 (+ 6384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:58917: BlockingRecv error: recv got EOF from 127.1.157.194:58917 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Dec 14 03:03:24 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:24 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.195@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:24.760551 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:24.756093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.195:47993)
1214 03:03:24.756379 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:24.756391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:24.756518 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:24.756754 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:24.756757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:24.756948 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:24.757134 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:24.757140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.757860 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:24.757863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:24.758225 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:24.758232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.758330 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:24.758822 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:24.758835 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:24.760431 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
W20251214 03:03:24.760640 11565 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:24.756168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:47993 (local address 127.1.157.254:44187)
1214 03:03:24.756322 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:24.756327 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:24.756458 (+ 131us) server_negotiation.cc:408] Connection header received
1214 03:03:24.756608 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:24.756612 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:24.756656 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:24.756719 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:24.757284 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.757762 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:24.758327 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:24.758504 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:24.760549 (+ 2045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.195:47993: BlockingRecv error: recv got EOF from 127.1.157.195:47993 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Dec 14 03:03:25 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:25 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.193@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:25.404745 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:25.399932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.193:35131)
1214 03:03:25.400233 (+ 301us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:25.400248 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:25.400360 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:25.400601 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:25.400603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:25.400776 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:25.400973 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:25.400979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:25.401783 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:25.401786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:25.402159 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:25.402165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:25.402256 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:25.402810 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:25.402823 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:25.404588 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
W20251214 03:03:25.404851 11569 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:25.400066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:35131 (local address 127.1.157.254:44187)
1214 03:03:25.400326 (+ 260us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:25.400329 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:25.400344 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:03:25.400442 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:25.400445 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:25.400488 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:25.400578 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:25.401112 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:25.401688 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:25.402280 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:25.402435 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:25.404736 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.193:35131: BlockingRecv error: recv got EOF from 127.1.157.193:35131 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
Dec 14 03:03:25 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
Dec 14 03:03:25 dist-test-slave-k7z7 krb5kdc[10502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.157.194@KRBTEST.COM for oryx/127.1.157.254@KRBTEST.COM, Server not found in Kerberos database
I20251214 03:03:25.603596 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:25.598912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:44187 (local address 127.1.157.194:58873)
1214 03:03:25.599176 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:25.599190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:25.599289 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:25.599500 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:25.599502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:25.599674 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:25.599871 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:25.599877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:25.600664 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:25.600667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:25.601046 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:25.601053 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:25.601152 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:25.601741 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:25.601754 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:25.603476 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.157.254:44187: Server oryx/127.1.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
W20251214 03:03:25.603729 11569 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:25.599023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:58873 (local address 127.1.157.254:44187)
1214 03:03:25.599180 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:25.599183 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:25.599267 (+ 84us) server_negotiation.cc:408] Connection header received
1214 03:03:25.599360 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:25.599362 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:25.599406 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:25.599468 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:25.600007 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:25.600550 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:25.601138 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:25.601328 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:25.603614 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.157.194:58873: BlockingRecv error: recv got EOF from 127.1.157.194:58873 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20251214 03:03:25.671744 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 11094
I20251214 03:03:25.676645 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 11246
I20251214 03:03:25.681381 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 11378
I20251214 03:03:25.685885 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 11024
2025-12-14T03:03:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18789 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-12-14T03:03:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:03:25Z Disabled control of system clock
I20251214 03:03:25.713222 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:42009
--webserver_interface=127.1.157.254
--webserver_port=0
--builtin_ntp_servers=127.1.157.212:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:42009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251214 03:03:25.819996 11574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:25.820217 11574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:25.820261 11574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:25.823469 11574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:03:25.823530 11574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:25.823547 11574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:25.823565 11574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:03:25.823580 11574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:03:25.827297 11574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:42009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:42009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:25.828363 11574 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:25.829507 11574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:25.835291 11579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:25.835357 11580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:25.835443 11582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:25.835336 11574 server_base.cc:1047] running on GCE node
I20251214 03:03:25.835997 11574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:25.836915 11574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:25.838056 11574 hybrid_clock.cc:648] HybridClock initialized: now 1765681405838040 us; error 37 us; skew 500 ppm
I20251214 03:03:25.839716 11574 webserver.cc:492] Webserver started at http://127.1.157.254:45473/ using document root <none> and password file <none>
I20251214 03:03:25.840243 11574 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:25.840307 11574 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:25.840478 11574 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:25.842181 11574 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "5fab6eaf9da14e4fba781196f69e083a"
format_stamp: "Formatted at 2025-12-14 03:03:25 on dist-test-slave-k7z7"
server_key: "2e6818183da7b126ee281af795b55676"
server_key_iv: "ef005b76bcbd23e091a8046416ac8334"
server_key_version: "encryptionkey@0"
I20251214 03:03:25.842640 11574 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "5fab6eaf9da14e4fba781196f69e083a"
format_stamp: "Formatted at 2025-12-14 03:03:25 on dist-test-slave-k7z7"
server_key: "2e6818183da7b126ee281af795b55676"
server_key_iv: "ef005b76bcbd23e091a8046416ac8334"
server_key_version: "encryptionkey@0"
I20251214 03:03:25.845804 11574 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:03:25.848130 11588 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:25.849486 11574 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20251214 03:03:25.849604 11574 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "5fab6eaf9da14e4fba781196f69e083a"
format_stamp: "Formatted at 2025-12-14 03:03:25 on dist-test-slave-k7z7"
server_key: "2e6818183da7b126ee281af795b55676"
server_key_iv: "ef005b76bcbd23e091a8046416ac8334"
server_key_version: "encryptionkey@0"
I20251214 03:03:25.849700 11574 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:25.866657 11574 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:25.867323 11574 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:25.867486 11574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:25.874577 11640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:42009 every 8 connection(s)
I20251214 03:03:25.874574 11574 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:42009
I20251214 03:03:25.875593 11574 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:03:25.878255 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 11574
I20251214 03:03:25.878371 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:03:25.878429 11641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:25.878647 1655 external_mini_cluster.cc:1468] Setting key 04423232178d9b0cc40230ddbf9f7c5c
I20251214 03:03:25.883960 11641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a: Bootstrap starting.
I20251214 03:03:25.885843 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:25.879859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.1.157.254:42009)
1214 03:03:25.880462 (+ 603us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:25.880470 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:25.880503 (+ 33us) server_negotiation.cc:408] Connection header received
1214 03:03:25.881150 (+ 647us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:25.881191 (+ 41us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:25.881549 (+ 358us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:25.881833 (+ 284us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:25.882288 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:25.883019 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:25.883637 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:25.883898 (+ 261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:25.884396 (+ 498us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:25.884436 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:25.884445 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:03:25.884784 (+ 339us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:25.884822 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:03:25.884836 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:25.884923 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:25.885068 (+ 145us) server_negotiation.cc:300] Negotiation successful
1214 03:03:25.885308 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20251214 03:03:25.886054 11641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:25.886843 11641 log.cc:826] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:25.888465 11641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a: No bootstrap required, opened a new log
I20251214 03:03:25.890870 11641 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fab6eaf9da14e4fba781196f69e083a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42009 } }
I20251214 03:03:25.891036 11641 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:25.891110 11641 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fab6eaf9da14e4fba781196f69e083a, State: Initialized, Role: FOLLOWER
I20251214 03:03:25.891480 11641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fab6eaf9da14e4fba781196f69e083a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42009 } }
I20251214 03:03:25.891597 11641 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:03:25.891649 11641 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:03:25.891736 11641 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:25.892452 11641 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fab6eaf9da14e4fba781196f69e083a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42009 } }
I20251214 03:03:25.892748 11641 leader_election.cc:304] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fab6eaf9da14e4fba781196f69e083a; no voters:
I20251214 03:03:25.893004 11641 leader_election.cc:290] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:03:25.893096 11646 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:25.893314 11646 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [term 1 LEADER]: Becoming Leader. State: Replica: 5fab6eaf9da14e4fba781196f69e083a, State: Running, Role: LEADER
I20251214 03:03:25.893631 11646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fab6eaf9da14e4fba781196f69e083a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42009 } }
I20251214 03:03:25.893998 11641 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:03:25.894981 11648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fab6eaf9da14e4fba781196f69e083a. Latest consensus state: current_term: 1 leader_uuid: "5fab6eaf9da14e4fba781196f69e083a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fab6eaf9da14e4fba781196f69e083a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42009 } } }
I20251214 03:03:25.895102 11648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:25.895462 11655 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:03:25.895601 11647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fab6eaf9da14e4fba781196f69e083a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fab6eaf9da14e4fba781196f69e083a" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 42009 } } }
I20251214 03:03:25.895689 11647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:25.897917 11655 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:03:25.902618 11655 catalog_manager.cc:1357] Generated new cluster ID: c882213eb525463b9e5f4740f98e628d
I20251214 03:03:25.902670 11655 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:03:25.928079 11655 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:03:25.929111 11655 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:03:25.938764 11655 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5fab6eaf9da14e4fba781196f69e083a: Generated new TSK 0
I20251214 03:03:25.939512 11655 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 03:03:25.953766 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42009
--builtin_ntp_servers=127.1.157.212:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251214 03:03:26.058580 11665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:26.058780 11665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:26.058828 11665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:26.062155 11665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:26.062228 11665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:26.062310 11665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:03:26.066308 11665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:26.067461 11665 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:26.069068 11665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:26.076131 11673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:26.076123 11670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:26.076115 11671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:26.076380 11665 server_base.cc:1047] running on GCE node
I20251214 03:03:26.076835 11665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:26.077471 11665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:26.078668 11665 hybrid_clock.cc:648] HybridClock initialized: now 1765681406078640 us; error 46 us; skew 500 ppm
I20251214 03:03:26.080621 11665 webserver.cc:492] Webserver started at http://127.1.157.193:44207/ using document root <none> and password file <none>
I20251214 03:03:26.081210 11665 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:26.081313 11665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:26.081560 11665 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:26.083276 11665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "21e05b791500437db07a274aa50b529c"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "469d41bba380433b7b9dfbfc0a084fa9"
server_key_iv: "6464af2fc917b27a5f3ad21839b7915c"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.083739 11665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "21e05b791500437db07a274aa50b529c"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "469d41bba380433b7b9dfbfc0a084fa9"
server_key_iv: "6464af2fc917b27a5f3ad21839b7915c"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.087024 11665 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:03:26.089205 11679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.090160 11665 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251214 03:03:26.090265 11665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "21e05b791500437db07a274aa50b529c"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "469d41bba380433b7b9dfbfc0a084fa9"
server_key_iv: "6464af2fc917b27a5f3ad21839b7915c"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.090348 11665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:26.107837 11665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:26.108441 11665 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:26.108596 11665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:26.109118 11665 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:26.110123 11665 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:26.110172 11665 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.110230 11665 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:26.110262 11665 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.119059 11665 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:43737
I20251214 03:03:26.119088 11792 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:43737 every 8 connection(s)
I20251214 03:03:26.120110 11665 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:03:26.127911 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:26.121870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:54391 (local address 127.1.157.254:42009)
1214 03:03:26.121992 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:26.121996 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:26.122893 (+ 897us) server_negotiation.cc:408] Connection header received
1214 03:03:26.123744 (+ 851us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:26.123747 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:26.123809 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:26.123881 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:26.124843 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.125404 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:26.126034 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.126209 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:26.127420 (+ 1211us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:26.127440 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:26.127443 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:03:26.127456 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:26.127475 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:03:26.127482 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:26.127543 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:26.127693 (+ 150us) server_negotiation.cc:300] Negotiation successful
1214 03:03:26.127762 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251214 03:03:26.128600 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:26.122148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42009 (local address 127.1.157.193:54391)
1214 03:03:26.122761 (+ 613us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:26.122794 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:26.123547 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:26.124048 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:26.124056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:26.124134 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:26.124701 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:26.124713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.125523 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:26.125532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:26.125925 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:26.125932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.126143 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:26.127207 (+ 1064us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:03:26.127208 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:26.127240 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:03:26.127295 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:26.127560 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:26.127568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:26.127572 (+ 4us) client_negotiation.cc:769] Sending connection context
1214 03:03:26.127693 (+ 121us) client_negotiation.cc:240] Negotiation successful
1214 03:03:26.127924 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":92,"threads_started":1}
I20251214 03:03:26.129675 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 11665
I20251214 03:03:26.129776 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:03:26.129792 11793 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42009
I20251214 03:03:26.130033 11793 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:26.130050 1655 external_mini_cluster.cc:1468] Setting key 6cb76b9189aa691151b7d1d620226583
I20251214 03:03:26.130648 11793 heartbeater.cc:507] Master 127.1.157.254:42009 requested a full tablet report, sending...
I20251214 03:03:26.131997 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42009
--builtin_ntp_servers=127.1.157.212:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251214 03:03:26.132073 11605 ts_manager.cc:194] Registered new tserver with Master: 21e05b791500437db07a274aa50b529c (127.1.157.193:43737)
I20251214 03:03:26.133450 11605 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.157.193:54391
W20251214 03:03:26.236018 11796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:26.236213 11796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:26.236256 11796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:26.239411 11796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:26.239459 11796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:26.239526 11796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:03:26.243227 11796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:26.244168 11796 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:26.245162 11796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:26.251279 11804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:26.251385 11802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:26.251297 11801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:26.251529 11796 server_base.cc:1047] running on GCE node
I20251214 03:03:26.251866 11796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:26.252368 11796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:26.253496 11796 hybrid_clock.cc:648] HybridClock initialized: now 1765681406253481 us; error 29 us; skew 500 ppm
I20251214 03:03:26.255159 11796 webserver.cc:492] Webserver started at http://127.1.157.194:33261/ using document root <none> and password file <none>
I20251214 03:03:26.255673 11796 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:26.255722 11796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:26.255864 11796 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:26.257509 11796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "df6431a05ec8413d9e99c3f7dfed6ca2"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "11b0a3b520670330f9a98e1d7658e098"
server_key_iv: "8f4ffa81b8b03d84b0d119e9527c2a95"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.257927 11796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "df6431a05ec8413d9e99c3f7dfed6ca2"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "11b0a3b520670330f9a98e1d7658e098"
server_key_iv: "8f4ffa81b8b03d84b0d119e9527c2a95"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.261021 11796 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:03:26.263216 11810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.264163 11796 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:26.264261 11796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "df6431a05ec8413d9e99c3f7dfed6ca2"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "11b0a3b520670330f9a98e1d7658e098"
server_key_iv: "8f4ffa81b8b03d84b0d119e9527c2a95"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.264358 11796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:26.280251 11796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:26.280838 11796 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:26.280993 11796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:26.281570 11796 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:26.282529 11796 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:26.282578 11796 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.282630 11796 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:26.282657 11796 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.290992 11796 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:39877
I20251214 03:03:26.291014 11923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:39877 every 8 connection(s)
I20251214 03:03:26.291934 11796 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:03:26.297655 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 11796
I20251214 03:03:26.297780 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:03:26.298018 1655 external_mini_cluster.cc:1468] Setting key 3b9a899f0a4d291ad383a4375c72cab2
I20251214 03:03:26.299667 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:26.293620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:44501 (local address 127.1.157.254:42009)
1214 03:03:26.293769 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:26.293774 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:26.294658 (+ 884us) server_negotiation.cc:408] Connection header received
1214 03:03:26.295484 (+ 826us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:26.295487 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:26.295542 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:26.295612 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:26.296557 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.297049 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:26.297705 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.297887 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:26.299155 (+ 1268us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:26.299172 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:26.299175 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:03:26.299191 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:26.299216 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:03:26.299226 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:26.299321 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:26.299497 (+ 176us) server_negotiation.cc:300] Negotiation successful
1214 03:03:26.299558 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251214 03:03:26.300016 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42009
--builtin_ntp_servers=127.1.157.212:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251214 03:03:26.300516 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:26.293921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42009 (local address 127.1.157.194:44501)
1214 03:03:26.294525 (+ 604us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:26.294563 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:26.295264 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:26.295750 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:26.295757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:26.295833 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:26.296434 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:26.296447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.297152 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:26.297155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:26.297587 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:26.297595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.297811 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:26.298942 (+ 1131us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:03:26.298944 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:26.298976 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:03:26.299030 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:26.299331 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:26.299340 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:26.299346 (+ 6us) client_negotiation.cc:769] Sending connection context
1214 03:03:26.299499 (+ 153us) client_negotiation.cc:240] Negotiation successful
1214 03:03:26.299743 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
I20251214 03:03:26.301988 11924 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42009
I20251214 03:03:26.302214 11924 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:26.302755 11924 heartbeater.cc:507] Master 127.1.157.254:42009 requested a full tablet report, sending...
I20251214 03:03:26.303776 11605 ts_manager.cc:194] Registered new tserver with Master: df6431a05ec8413d9e99c3f7dfed6ca2 (127.1.157.194:39877)
I20251214 03:03:26.304513 11605 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.157.194:44501
W20251214 03:03:26.406059 11927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:26.406250 11927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:26.406291 11927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:26.409423 11927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:26.409472 11927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:26.409543 11927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:03:26.413239 11927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:26.414184 11927 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:26.415194 11927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:26.421751 11933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:26.421830 11927 server_base.cc:1047] running on GCE node
W20251214 03:03:26.421753 11935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:26.421753 11932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:26.422453 11927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:26.423038 11927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:26.424222 11927 hybrid_clock.cc:648] HybridClock initialized: now 1765681406424199 us; error 39 us; skew 500 ppm
I20251214 03:03:26.426232 11927 webserver.cc:492] Webserver started at http://127.1.157.195:35231/ using document root <none> and password file <none>
I20251214 03:03:26.427098 11927 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:26.427162 11927 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:26.427356 11927 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:26.429143 11927 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "4aa2ffd239844bacbb9bc5970cf3a209"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "f59dfaa198888c0233f78d6703f4e43b"
server_key_iv: "50af94207887d041ecdf3cd1cb767876"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.429697 11927 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "4aa2ffd239844bacbb9bc5970cf3a209"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "f59dfaa198888c0233f78d6703f4e43b"
server_key_iv: "50af94207887d041ecdf3cd1cb767876"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.433663 11927 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20251214 03:03:26.436722 11941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.438280 11927 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:03:26.438424 11927 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "4aa2ffd239844bacbb9bc5970cf3a209"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "f59dfaa198888c0233f78d6703f4e43b"
server_key_iv: "50af94207887d041ecdf3cd1cb767876"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.438536 11927 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:26.457980 11927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:26.458647 11927 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:26.458811 11927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:26.459375 11927 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:26.460418 11927 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:26.460469 11927 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.460523 11927 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:26.460557 11927 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.469295 11927 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:45863
I20251214 03:03:26.469323 12054 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:45863 every 8 connection(s)
I20251214 03:03:26.470227 11927 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:03:26.476055 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 11927
I20251214 03:03:26.476161 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:03:26.476393 1655 external_mini_cluster.cc:1468] Setting key dfb7d08bb2a2a62819dda74d29dece11
I20251214 03:03:26.477985 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:26.472029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:39703 (local address 127.1.157.254:42009)
1214 03:03:26.472205 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:26.472210 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:26.473075 (+ 865us) server_negotiation.cc:408] Connection header received
1214 03:03:26.473891 (+ 816us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:26.473894 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:26.473944 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:26.474016 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:26.474928 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.475433 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:26.476068 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.476261 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:26.477514 (+ 1253us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:26.477532 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:26.477535 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:03:26.477549 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:26.477570 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:03:26.477577 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:26.477638 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:26.477782 (+ 144us) server_negotiation.cc:300] Negotiation successful
1214 03:03:26.477842 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251214 03:03:26.478646 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:26.472330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:42009 (local address 127.1.157.195:39703)
1214 03:03:26.472943 (+ 613us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:26.472976 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:26.473716 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:26.474151 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:26.474159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:26.474234 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:26.474795 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:26.474807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.475546 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:26.475549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:26.475955 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:26.475963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:26.476161 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:26.477306 (+ 1145us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:03:26.477307 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:26.477339 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:03:26.477391 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:26.477649 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:26.477658 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:26.477663 (+ 5us) client_negotiation.cc:769] Sending connection context
1214 03:03:26.477781 (+ 118us) client_negotiation.cc:240] Negotiation successful
1214 03:03:26.478004 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
I20251214 03:03:26.479585 12055 heartbeater.cc:344] Connected to a master server at 127.1.157.254:42009
I20251214 03:03:26.479836 12055 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:26.480406 12055 heartbeater.cc:507] Master 127.1.157.254:42009 requested a full tablet report, sending...
I20251214 03:03:26.481395 11605 ts_manager.cc:194] Registered new tserver with Master: 4aa2ffd239844bacbb9bc5970cf3a209 (127.1.157.195:45863)
I20251214 03:03:26.481953 11605 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.157.195:39703
I20251214 03:03:26.490126 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251214 03:03:26.492638 11644 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:26.491939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.1.157.254:42009)
1214 03:03:26.492049 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:26.492053 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:26.492117 (+ 64us) server_negotiation.cc:408] Connection header received
1214 03:03:26.492231 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:26.492234 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:26.492279 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:26.492338 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:26.492541 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43550: BlockingRecv error: recv got EOF from 127.0.0.1:43550 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20251214 03:03:26.492782 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 11665
I20251214 03:03:26.497639 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 11796
I20251214 03:03:26.502213 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 11927
I20251214 03:03:26.506691 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 11574
2025-12-14T03:03:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (819 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-12-14T03:03:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:03:26Z Disabled control of system clock
I20251214 03:03:26.532905 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:45231
--webserver_interface=127.1.157.254
--webserver_port=0
--builtin_ntp_servers=127.1.157.212:46177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:45231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251214 03:03:26.636333 12066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:26.636583 12066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:26.636641 12066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:26.640012 12066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:03:26.640079 12066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:26.640094 12066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:26.640112 12066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:03:26.640129 12066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:03:26.644384 12066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:46177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:45231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:45231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:26.645509 12066 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:26.646582 12066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:26.652138 12072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:26.652086 12074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:26.652108 12071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:26.652396 12066 server_base.cc:1047] running on GCE node
I20251214 03:03:26.652885 12066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:26.653827 12066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:26.654990 12066 hybrid_clock.cc:648] HybridClock initialized: now 1765681406654978 us; error 26 us; skew 500 ppm
I20251214 03:03:26.656985 12066 webserver.cc:492] Webserver started at http://127.1.157.254:39607/ using document root <none> and password file <none>
I20251214 03:03:26.657632 12066 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:26.657701 12066 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:26.657904 12066 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:26.659726 12066 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "38b43f391ab94a56ac55f398692de9ba"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "54e68078dca07282a1db4bd5457b44f2"
server_key_iv: "95eff59cf2635417c4be12358b9a5b4f"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.660241 12066 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "38b43f391ab94a56ac55f398692de9ba"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "54e68078dca07282a1db4bd5457b44f2"
server_key_iv: "95eff59cf2635417c4be12358b9a5b4f"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.663857 12066 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251214 03:03:26.666209 12080 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.667301 12066 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251214 03:03:26.667423 12066 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "38b43f391ab94a56ac55f398692de9ba"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "54e68078dca07282a1db4bd5457b44f2"
server_key_iv: "95eff59cf2635417c4be12358b9a5b4f"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.667518 12066 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:26.672226 12066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:26.672775 12066 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:26.672917 12066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:26.680969 12132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:45231 every 8 connection(s)
I20251214 03:03:26.680969 12066 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:45231
I20251214 03:03:26.682121 12066 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:03:26.684952 12133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:26.687697 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 12066
I20251214 03:03:26.687816 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:03:26.688043 1655 external_mini_cluster.cc:1468] Setting key 7eccaa52f68a58a88bf161ff6f516ed8
I20251214 03:03:26.691720 12133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba: Bootstrap starting.
I20251214 03:03:26.693276 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:26.689297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.1.157.254:45231)
1214 03:03:26.689947 (+ 650us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:26.689956 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:26.689992 (+ 36us) server_negotiation.cc:408] Connection header received
1214 03:03:26.690637 (+ 645us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:26.690661 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:26.690994 (+ 333us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:26.691319 (+ 325us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:26.692153 (+ 834us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:26.692159 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:26.692169 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:03:26.692180 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:26.692221 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:03:26.692234 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:26.692366 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:26.692450 (+ 84us) server_negotiation.cc:300] Negotiation successful
1214 03:03:26.692665 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20251214 03:03:26.694775 12133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:26.695609 12133 log.cc:826] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:26.697556 12133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba: No bootstrap required, opened a new log
I20251214 03:03:26.700531 12133 raft_consensus.cc:359] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b43f391ab94a56ac55f398692de9ba" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 45231 } }
I20251214 03:03:26.700736 12133 raft_consensus.cc:385] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:26.700790 12133 raft_consensus.cc:740] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b43f391ab94a56ac55f398692de9ba, State: Initialized, Role: FOLLOWER
I20251214 03:03:26.701252 12133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b43f391ab94a56ac55f398692de9ba" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 45231 } }
I20251214 03:03:26.701396 12133 raft_consensus.cc:399] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:03:26.701462 12133 raft_consensus.cc:493] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:03:26.701552 12133 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:26.702445 12133 raft_consensus.cc:515] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b43f391ab94a56ac55f398692de9ba" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 45231 } }
I20251214 03:03:26.702818 12133 leader_election.cc:304] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b43f391ab94a56ac55f398692de9ba; no voters:
I20251214 03:03:26.703163 12133 leader_election.cc:290] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:03:26.703264 12138 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:26.703419 12138 raft_consensus.cc:697] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [term 1 LEADER]: Becoming Leader. State: Replica: 38b43f391ab94a56ac55f398692de9ba, State: Running, Role: LEADER
I20251214 03:03:26.703718 12138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b43f391ab94a56ac55f398692de9ba" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 45231 } }
I20251214 03:03:26.704401 12133 sys_catalog.cc:565] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:03:26.705411 12139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38b43f391ab94a56ac55f398692de9ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b43f391ab94a56ac55f398692de9ba" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 45231 } } }
I20251214 03:03:26.705549 12139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:26.705964 12140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b43f391ab94a56ac55f398692de9ba. Latest consensus state: current_term: 1 leader_uuid: "38b43f391ab94a56ac55f398692de9ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b43f391ab94a56ac55f398692de9ba" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 45231 } } }
I20251214 03:03:26.706071 12140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:26.705950 12147 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:03:26.709317 12147 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:03:26.714574 12147 catalog_manager.cc:1357] Generated new cluster ID: 18b0b2d705be48c397210f2431d7430f
I20251214 03:03:26.714641 12147 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:03:26.734346 12147 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:03:26.734699 12147 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:03:26.747293 12147 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 38b43f391ab94a56ac55f398692de9ba: Generated new TSK 0
I20251214 03:03:26.748072 12147 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 03:03:26.759265 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:45231
--builtin_ntp_servers=127.1.157.212:46177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251214 03:03:26.867566 12157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:26.867786 12157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:26.867849 12157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:26.871142 12157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:26.871205 12157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:26.871302 12157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:03:26.875094 12157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:46177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:45231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:26.876065 12157 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:26.877116 12157 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:26.883440 12163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:26.883406 12162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:26.883560 12157 server_base.cc:1047] running on GCE node
W20251214 03:03:26.883679 12165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:26.883996 12157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:26.884567 12157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:26.885716 12157 hybrid_clock.cc:648] HybridClock initialized: now 1765681406885694 us; error 34 us; skew 500 ppm
I20251214 03:03:26.887355 12157 webserver.cc:492] Webserver started at http://127.1.157.193:36485/ using document root <none> and password file <none>
I20251214 03:03:26.887861 12157 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:26.887912 12157 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:26.888072 12157 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:26.889690 12157 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "e7c2be56e8894ad6a0df55aa46ccb2a8"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "cc2baf29cd777acbfc0179889ef9e72e"
server_key_iv: "15f691c64443f5abb7996ba9854b1c02"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.890113 12157 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "e7c2be56e8894ad6a0df55aa46ccb2a8"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "cc2baf29cd777acbfc0179889ef9e72e"
server_key_iv: "15f691c64443f5abb7996ba9854b1c02"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.893558 12157 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:03:26.895607 12171 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.896561 12157 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:26.896653 12157 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "e7c2be56e8894ad6a0df55aa46ccb2a8"
format_stamp: "Formatted at 2025-12-14 03:03:26 on dist-test-slave-k7z7"
server_key: "cc2baf29cd777acbfc0179889ef9e72e"
server_key_iv: "15f691c64443f5abb7996ba9854b1c02"
server_key_version: "encryptionkey@0"
I20251214 03:03:26.896744 12157 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:26.901152 12157 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:26.901710 12157 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:26.901845 12157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:26.902323 12157 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:26.903170 12157 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:26.903223 12157 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.903257 12157 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:26.903272 12157 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:26.911883 12157 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:35413
I20251214 03:03:26.911907 12284 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:35413 every 8 connection(s)
I20251214 03:03:26.912801 12157 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:03:26.914786 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 12157
I20251214 03:03:26.914916 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:03:26.915185 1655 external_mini_cluster.cc:1468] Setting key e6018503e75d50e1d62b53a2b4d3cd04
I20251214 03:03:26.917115 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:45231
--builtin_ntp_servers=127.1.157.212:46177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20251214 03:03:26.918078 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:26.914427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:47819 (local address 127.1.157.254:45231)
1214 03:03:26.914561 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:26.914565 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:26.915502 (+ 937us) server_negotiation.cc:408] Connection header received
1214 03:03:26.916395 (+ 893us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:26.916400 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:26.916441 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:26.916504 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:26.917672 (+ 1168us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:26.917676 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:26.917678 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:03:26.917692 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:26.917716 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:03:26.917725 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:26.917807 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:26.917926 (+ 119us) server_negotiation.cc:300] Negotiation successful
1214 03:03:26.917962 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251214 03:03:26.918744 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:26.914706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:45231 (local address 127.1.157.193:47819)
1214 03:03:26.915365 (+ 659us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:26.915401 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:26.916178 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:26.916663 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:26.916671 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:26.916751 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:26.917421 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:03:26.917422 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:26.917470 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:03:26.917524 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:26.917808 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:26.917813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:26.917816 (+ 3us) client_negotiation.cc:769] Sending connection context
1214 03:03:26.917964 (+ 148us) client_negotiation.cc:240] Negotiation successful
1214 03:03:26.918122 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20251214 03:03:26.919799 12285 heartbeater.cc:344] Connected to a master server at 127.1.157.254:45231
I20251214 03:03:26.920006 12285 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:26.920411 12285 heartbeater.cc:507] Master 127.1.157.254:45231 requested a full tablet report, sending...
I20251214 03:03:26.921753 12097 ts_manager.cc:194] Registered new tserver with Master: e7c2be56e8894ad6a0df55aa46ccb2a8 (127.1.157.193:35413)
W20251214 03:03:27.025537 12288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:27.025753 12288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:27.025823 12288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:27.029156 12288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:27.029256 12288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:27.029356 12288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:03:27.033270 12288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:46177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:45231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:27.034366 12288 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:27.035439 12288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:27.041893 12293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:27.041985 12288 server_base.cc:1047] running on GCE node
W20251214 03:03:27.041908 12296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:27.041906 12294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:27.042495 12288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:27.042986 12288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:27.044137 12288 hybrid_clock.cc:648] HybridClock initialized: now 1765681407044125 us; error 23 us; skew 500 ppm
I20251214 03:03:27.045848 12288 webserver.cc:492] Webserver started at http://127.1.157.194:36101/ using document root <none> and password file <none>
I20251214 03:03:27.046368 12288 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:27.046419 12288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:27.046604 12288 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:27.048215 12288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "3783fe58fa3d4e63904f66d3c81cb65d"
format_stamp: "Formatted at 2025-12-14 03:03:27 on dist-test-slave-k7z7"
server_key: "89b53243b1acc0d96295478abca85e43"
server_key_iv: "4e258aa3d0b84a82a91a5e83eab90bac"
server_key_version: "encryptionkey@0"
I20251214 03:03:27.048661 12288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "3783fe58fa3d4e63904f66d3c81cb65d"
format_stamp: "Formatted at 2025-12-14 03:03:27 on dist-test-slave-k7z7"
server_key: "89b53243b1acc0d96295478abca85e43"
server_key_iv: "4e258aa3d0b84a82a91a5e83eab90bac"
server_key_version: "encryptionkey@0"
I20251214 03:03:27.051934 12288 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20251214 03:03:27.054045 12302 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:27.054981 12288 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20251214 03:03:27.055079 12288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "3783fe58fa3d4e63904f66d3c81cb65d"
format_stamp: "Formatted at 2025-12-14 03:03:27 on dist-test-slave-k7z7"
server_key: "89b53243b1acc0d96295478abca85e43"
server_key_iv: "4e258aa3d0b84a82a91a5e83eab90bac"
server_key_version: "encryptionkey@0"
I20251214 03:03:27.055160 12288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:27.059659 12288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:27.060164 12288 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:27.060312 12288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:27.060763 12288 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:27.061648 12288 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:27.061694 12288 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:27.061740 12288 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:27.061765 12288 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:27.070300 12288 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:44005
I20251214 03:03:27.070319 12415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:44005 every 8 connection(s)
I20251214 03:03:27.071344 12288 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:03:27.072721 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 12288
I20251214 03:03:27.072844 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:03:27.073117 1655 external_mini_cluster.cc:1468] Setting key a39f18699b86eaf348bf6da096827469
I20251214 03:03:27.075268 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:45231
--builtin_ntp_servers=127.1.157.212:46177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20251214 03:03:27.077214 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:27.073374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:45651 (local address 127.1.157.254:45231)
1214 03:03:27.073628 (+ 254us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:27.073634 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:27.074559 (+ 925us) server_negotiation.cc:408] Connection header received
1214 03:03:27.075526 (+ 967us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:27.075529 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:27.075569 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:27.075633 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:27.076746 (+ 1113us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:27.076751 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:27.076754 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:03:27.076768 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:27.076793 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:03:27.076803 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:27.076892 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:27.077008 (+ 116us) server_negotiation.cc:300] Negotiation successful
1214 03:03:27.077048 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20251214 03:03:27.077888 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:27.073680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:45231 (local address 127.1.157.194:45651)
1214 03:03:27.074421 (+ 741us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:27.074462 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:27.075287 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:27.075799 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:27.075809 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:27.075912 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:27.076464 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:03:27.076467 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:27.076514 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:03:27.076584 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:27.076890 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:27.076895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:27.076899 (+ 4us) client_negotiation.cc:769] Sending connection context
1214 03:03:27.077051 (+ 152us) client_negotiation.cc:240] Negotiation successful
1214 03:03:27.077248 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":113,"threads_started":1}
I20251214 03:03:27.078919 12416 heartbeater.cc:344] Connected to a master server at 127.1.157.254:45231
I20251214 03:03:27.079190 12416 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:27.079699 12416 heartbeater.cc:507] Master 127.1.157.254:45231 requested a full tablet report, sending...
I20251214 03:03:27.080888 12097 ts_manager.cc:194] Registered new tserver with Master: 3783fe58fa3d4e63904f66d3c81cb65d (127.1.157.194:44005)
W20251214 03:03:27.181996 12419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:27.182248 12419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:27.182301 12419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:27.185762 12419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:27.185833 12419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:27.185914 12419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:03:27.189916 12419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:46177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:45231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:27.191046 12419 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:27.192185 12419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:27.198571 12425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:27.198592 12424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:27.198592 12427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:27.198770 12419 server_base.cc:1047] running on GCE node
I20251214 03:03:27.199141 12419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:27.199646 12419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:27.200783 12419 hybrid_clock.cc:648] HybridClock initialized: now 1765681407200766 us; error 30 us; skew 500 ppm
I20251214 03:03:27.202453 12419 webserver.cc:492] Webserver started at http://127.1.157.195:33185/ using document root <none> and password file <none>
I20251214 03:03:27.202960 12419 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:27.203007 12419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:27.203163 12419 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:27.204762 12419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "9abbc371ef704dabb099572eb8bdee44"
format_stamp: "Formatted at 2025-12-14 03:03:27 on dist-test-slave-k7z7"
server_key: "17a9233eea4ca95e4033b31e1b907393"
server_key_iv: "0f7a882ddf3d9a60264a0a4c17149094"
server_key_version: "encryptionkey@0"
I20251214 03:03:27.205245 12419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "9abbc371ef704dabb099572eb8bdee44"
format_stamp: "Formatted at 2025-12-14 03:03:27 on dist-test-slave-k7z7"
server_key: "17a9233eea4ca95e4033b31e1b907393"
server_key_iv: "0f7a882ddf3d9a60264a0a4c17149094"
server_key_version: "encryptionkey@0"
I20251214 03:03:27.208479 12419 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20251214 03:03:27.210716 12433 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:27.211665 12419 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 03:03:27.211766 12419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "9abbc371ef704dabb099572eb8bdee44"
format_stamp: "Formatted at 2025-12-14 03:03:27 on dist-test-slave-k7z7"
server_key: "17a9233eea4ca95e4033b31e1b907393"
server_key_iv: "0f7a882ddf3d9a60264a0a4c17149094"
server_key_version: "encryptionkey@0"
I20251214 03:03:27.211848 12419 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:27.216343 12419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:27.216866 12419 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:27.217007 12419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:27.217533 12419 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:27.218436 12419 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:27.218482 12419 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:27.218529 12419 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:27.218556 12419 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:27.227172 12419 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:40931
I20251214 03:03:27.227195 12546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:40931 every 8 connection(s)
I20251214 03:03:27.228112 12419 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:03:27.231076 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 12419
I20251214 03:03:27.231182 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:03:27.231411 1655 external_mini_cluster.cc:1468] Setting key 3d830914c06683746a19993431ba59b9
I20251214 03:03:27.233379 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:27.229868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:57387 (local address 127.1.157.254:45231)
1214 03:03:27.230009 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:27.230013 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:27.230906 (+ 893us) server_negotiation.cc:408] Connection header received
1214 03:03:27.231753 (+ 847us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:27.231756 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:27.231796 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:27.231886 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:27.232930 (+ 1044us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:27.232933 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:27.232936 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1214 03:03:27.232946 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:27.232969 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1214 03:03:27.232976 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:27.233048 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:27.233210 (+ 162us) server_negotiation.cc:300] Negotiation successful
1214 03:03:27.233255 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251214 03:03:27.234036 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:27.230104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:45231 (local address 127.1.157.195:57387)
1214 03:03:27.230763 (+ 659us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:27.230798 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:27.231556 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:27.232015 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:27.232023 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:27.232105 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:27.232711 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1214 03:03:27.232713 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:27.232743 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1214 03:03:27.232795 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:27.233046 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:27.233053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:27.233058 (+ 5us) client_negotiation.cc:769] Sending connection context
1214 03:03:27.233271 (+ 213us) client_negotiation.cc:240] Negotiation successful
1214 03:03:27.233472 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":107,"threads_started":1}
I20251214 03:03:27.234901 12547 heartbeater.cc:344] Connected to a master server at 127.1.157.254:45231
I20251214 03:03:27.235105 12547 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:27.235486 12547 heartbeater.cc:507] Master 127.1.157.254:45231 requested a full tablet report, sending...
I20251214 03:03:27.236342 12097 ts_manager.cc:194] Registered new tserver with Master: 9abbc371ef704dabb099572eb8bdee44 (127.1.157.195:40931)
I20251214 03:03:27.244894 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251214 03:03:27.247400 12136 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:27.246631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.1.157.254:45231)
1214 03:03:27.246757 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:27.246762 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:27.246859 (+ 97us) server_negotiation.cc:408] Connection header received
1214 03:03:27.246966 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:27.246968 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:27.247005 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:27.247076 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:27.247287 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39172: BlockingRecv error: recv got EOF from 127.0.0.1:39172 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20251214 03:03:27.247493 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 12157
I20251214 03:03:27.252581 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 12288
I20251214 03:03:27.257503 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 12419
I20251214 03:03:27.261986 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 12066
2025-12-14T03:03:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (755 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:03:27 dist-test-slave-k7z7 krb5kdc[12557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:03:27 dist-test-slave-k7z7 krb5kdc[12557](info): set up 2 sockets
Dec 14 03:03:27 dist-test-slave-k7z7 krb5kdc[12557](info): commencing operation
krb5kdc: starting...
W20251214 03:03:29.109668 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.827s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:03:29 dist-test-slave-k7z7 krb5kdc[12557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:03:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:03:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:03:29.246145 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:33113
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:33113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:29.351226 12573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:29.351467 12573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:29.351545 12573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:29.354939 12573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:03:29.355010 12573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:29.355041 12573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:29.355067 12573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:03:29.355099 12573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:03:29.359458 12573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:33113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:29.360494 12573 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:29.361349 12573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:29.366680 12578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:29.366737 12579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:29.366683 12581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:29.366833 12573 server_base.cc:1047] running on GCE node
I20251214 03:03:29.367451 12573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:29.368299 12573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:29.369450 12573 hybrid_clock.cc:648] HybridClock initialized: now 1765681409369434 us; error 32 us; skew 500 ppm
Dec 14 03:03:29 dist-test-slave-k7z7 krb5kdc[12557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:29.371853 12573 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:03:29.372800 12573 webserver.cc:492] Webserver started at http://127.1.157.254:38679/ using document root <none> and password file <none>
I20251214 03:03:29.373323 12573 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:29.373373 12573 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:29.373528 12573 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:29.375102 12573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "ab08b7d88cb24738ab9ca2371b75e906"
format_stamp: "Formatted at 2025-12-14 03:03:29 on dist-test-slave-k7z7"
server_key: "bc17db25a7a2b9002cfcaf7d5fca4ada"
server_key_iv: "98b38462623f104017d9e7a3a3bcf105"
server_key_version: "encryptionkey@0"
I20251214 03:03:29.375516 12573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "ab08b7d88cb24738ab9ca2371b75e906"
format_stamp: "Formatted at 2025-12-14 03:03:29 on dist-test-slave-k7z7"
server_key: "bc17db25a7a2b9002cfcaf7d5fca4ada"
server_key_iv: "98b38462623f104017d9e7a3a3bcf105"
server_key_version: "encryptionkey@0"
I20251214 03:03:29.378813 12573 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251214 03:03:29.380973 12588 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:29.382007 12573 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:29.382121 12573 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "ab08b7d88cb24738ab9ca2371b75e906"
format_stamp: "Formatted at 2025-12-14 03:03:29 on dist-test-slave-k7z7"
server_key: "bc17db25a7a2b9002cfcaf7d5fca4ada"
server_key_iv: "98b38462623f104017d9e7a3a3bcf105"
server_key_version: "encryptionkey@0"
I20251214 03:03:29.382210 12573 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:29.398954 12573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:29.402122 12573 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:29.402314 12573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:29.409614 12573 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:33113
I20251214 03:03:29.409634 12640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:33113 every 8 connection(s)
I20251214 03:03:29.410636 12573 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:03:29.411767 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 12573
I20251214 03:03:29.411873 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:03:29.412117 1655 external_mini_cluster.cc:1468] Setting key 963df10f8d88932a06d6855775e060f0
I20251214 03:03:29.413650 12641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:29.418368 12641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906: Bootstrap starting.
Dec 14 03:03:29 dist-test-slave-k7z7 krb5kdc[12557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:03:29.420423 12641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:29.421156 12641 log.cc:826] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:29.423053 12641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906: No bootstrap required, opened a new log
I20251214 03:03:29.424141 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:29.413301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.1.157.254:33113)
1214 03:03:29.413733 (+ 432us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:29.413745 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:29.413788 (+ 43us) server_negotiation.cc:408] Connection header received
1214 03:03:29.414870 (+ 1082us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:29.414889 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:29.415171 (+ 282us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:29.415478 (+ 307us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:29.416326 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:29.417064 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:29.417709 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:29.417961 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:29.419953 (+ 1992us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:29.419971 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:29.419980 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:29.420012 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:29.421805 (+ 1793us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:29.422221 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:29.422225 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:29.422229 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:29.422316 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:29.422543 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:29.422545 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:29.422546 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:29.422835 (+ 289us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:29.422989 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:29.423329 (+ 340us) server_negotiation.cc:300] Negotiation successful
1214 03:03:29.423582 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20251214 03:03:29.425657 12641 raft_consensus.cc:359] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab08b7d88cb24738ab9ca2371b75e906" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33113 } }
I20251214 03:03:29.425860 12641 raft_consensus.cc:385] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:29.425930 12641 raft_consensus.cc:740] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab08b7d88cb24738ab9ca2371b75e906, State: Initialized, Role: FOLLOWER
I20251214 03:03:29.426329 12641 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab08b7d88cb24738ab9ca2371b75e906" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33113 } }
I20251214 03:03:29.426442 12641 raft_consensus.cc:399] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:03:29.426498 12641 raft_consensus.cc:493] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:03:29.426577 12641 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:29.427309 12641 raft_consensus.cc:515] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab08b7d88cb24738ab9ca2371b75e906" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33113 } }
I20251214 03:03:29.427606 12641 leader_election.cc:304] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab08b7d88cb24738ab9ca2371b75e906; no voters:
I20251214 03:03:29.427848 12641 leader_election.cc:290] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:03:29.428164 12646 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:29.428350 12646 raft_consensus.cc:697] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [term 1 LEADER]: Becoming Leader. State: Replica: ab08b7d88cb24738ab9ca2371b75e906, State: Running, Role: LEADER
I20251214 03:03:29.428608 12646 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab08b7d88cb24738ab9ca2371b75e906" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33113 } }
I20251214 03:03:29.428853 12641 sys_catalog.cc:565] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:03:29.430145 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab08b7d88cb24738ab9ca2371b75e906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab08b7d88cb24738ab9ca2371b75e906" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33113 } } }
I20251214 03:03:29.430164 12648 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab08b7d88cb24738ab9ca2371b75e906. Latest consensus state: current_term: 1 leader_uuid: "ab08b7d88cb24738ab9ca2371b75e906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab08b7d88cb24738ab9ca2371b75e906" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 33113 } } }
I20251214 03:03:29.430265 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:29.430267 12648 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906 [sys.catalog]: This master's current role is: LEADER
I20251214 03:03:29.430570 12655 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:03:29.433671 12655 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:03:29.438633 12655 catalog_manager.cc:1357] Generated new cluster ID: 1e23f9c8090440d08a0b331374d7801b
I20251214 03:03:29.438683 12655 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:03:29.462319 12655 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:03:29.463307 12655 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:03:29.471995 12655 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906: Generated new TSK 0
I20251214 03:03:29.472659 12655 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:03:29.526230 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:33113
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:29.659108 12669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:29.659356 12669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:29.659432 12669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:29.662843 12669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:29.662907 12669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:29.663004 12669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:03:29.667232 12669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:29.668215 12669 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:29.669003 12669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:29.675313 12677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:29.675380 12669 server_base.cc:1047] running on GCE node
W20251214 03:03:29.675309 12674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:29.675525 12675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:29.675894 12669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:29.676422 12669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:29.677583 12669 hybrid_clock.cc:648] HybridClock initialized: now 1765681409677560 us; error 39 us; skew 500 ppm
Dec 14 03:03:29 dist-test-slave-k7z7 krb5kdc[12557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:29.679965 12669 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:03:29.680914 12669 webserver.cc:492] Webserver started at http://127.1.157.193:37211/ using document root <none> and password file <none>
I20251214 03:03:29.681434 12669 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:29.681480 12669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:29.681643 12669 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:29.683182 12669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "0eb463dd87654688bdfac813d4a1c98a"
format_stamp: "Formatted at 2025-12-14 03:03:29 on dist-test-slave-k7z7"
server_key: "66e03f8c8728e146b75c441361cb6b38"
server_key_iv: "29a2580706e02b1a963c0169cee78bde"
server_key_version: "encryptionkey@0"
I20251214 03:03:29.683593 12669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "0eb463dd87654688bdfac813d4a1c98a"
format_stamp: "Formatted at 2025-12-14 03:03:29 on dist-test-slave-k7z7"
server_key: "66e03f8c8728e146b75c441361cb6b38"
server_key_iv: "29a2580706e02b1a963c0169cee78bde"
server_key_version: "encryptionkey@0"
I20251214 03:03:29.686833 12669 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251214 03:03:29.688863 12684 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:29.689836 12669 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:29.689927 12669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "0eb463dd87654688bdfac813d4a1c98a"
format_stamp: "Formatted at 2025-12-14 03:03:29 on dist-test-slave-k7z7"
server_key: "66e03f8c8728e146b75c441361cb6b38"
server_key_iv: "29a2580706e02b1a963c0169cee78bde"
server_key_version: "encryptionkey@0"
I20251214 03:03:29.690018 12669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:29.706132 12669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:29.712913 12669 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:29.713081 12669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:29.713649 12669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:29.714510 12669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:29.714555 12669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:29.714597 12669 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:29.714627 12669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:29.722908 12669 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:37835
I20251214 03:03:29.722981 12797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:37835 every 8 connection(s)
I20251214 03:03:29.723807 12669 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
Dec 14 03:03:29 dist-test-slave-k7z7 krb5kdc[12557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:03:29.732753 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 12669
I20251214 03:03:29.732882 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:03:29.733111 1655 external_mini_cluster.cc:1468] Setting key 4cca15a6ad02cb6c9d766e394be14112
I20251214 03:03:29.735183 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:29.725434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:57891 (local address 127.1.157.254:33113)
1214 03:03:29.725556 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:29.725560 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:29.726183 (+ 623us) server_negotiation.cc:408] Connection header received
1214 03:03:29.726954 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:29.726957 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:29.727016 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:29.727085 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:29.728637 (+ 1552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:29.729126 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:29.729747 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:29.729922 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:29.731951 (+ 2029us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:29.731971 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:29.731973 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:29.731997 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:29.733392 (+ 1395us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:29.733874 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:29.733877 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:29.733878 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:29.733920 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:29.734190 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:29.734193 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:29.734195 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:29.734361 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:29.734445 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:29.734914 (+ 469us) server_negotiation.cc:300] Negotiation successful
1214 03:03:29.735038 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251214 03:03:29.735939 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:29.725663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33113 (local address 127.1.157.193:57891)
1214 03:03:29.726056 (+ 393us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:29.726090 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:29.726773 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:29.727239 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:29.727246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:29.727611 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:29.728494 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:29.728509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:29.729255 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:29.729258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:29.729642 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:29.729649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:29.729843 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:29.730431 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:29.730454 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:29.731828 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:29.733506 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:29.733511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:29.733521 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:29.733787 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:29.734000 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:29.734003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:29.734005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:29.734110 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:29.734459 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:29.734466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:29.734676 (+ 210us) client_negotiation.cc:769] Sending connection context
1214 03:03:29.734845 (+ 169us) client_negotiation.cc:240] Negotiation successful
1214 03:03:29.735081 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20251214 03:03:29.736937 12798 heartbeater.cc:344] Connected to a master server at 127.1.157.254:33113
I20251214 03:03:29.737195 12798 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:29.737754 12798 heartbeater.cc:507] Master 127.1.157.254:33113 requested a full tablet report, sending...
I20251214 03:03:29.739145 12605 ts_manager.cc:194] Registered new tserver with Master: 0eb463dd87654688bdfac813d4a1c98a (127.1.157.193:37835)
I20251214 03:03:29.740209 12605 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:57891
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:03:29.776166 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:33113
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:29.878460 12805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:29.878657 12805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:29.878711 12805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:29.882228 12805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:29.882293 12805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:29.882369 12805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:03:29.886379 12805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:29.887311 12805 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:29.888072 12805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:29.894047 12810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:29.894146 12813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:29.894150 12811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:29.894413 12805 server_base.cc:1047] running on GCE node
I20251214 03:03:29.894773 12805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:29.895277 12805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:29.896418 12805 hybrid_clock.cc:648] HybridClock initialized: now 1765681409896396 us; error 36 us; skew 500 ppm
Dec 14 03:03:29 dist-test-slave-k7z7 krb5kdc[12557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:29.898808 12805 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:03:29.899782 12805 webserver.cc:492] Webserver started at http://127.1.157.194:45113/ using document root <none> and password file <none>
I20251214 03:03:29.900270 12805 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:29.900312 12805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:29.900457 12805 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:29.902102 12805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "cb665290b4114b078b3834d543031cd9"
format_stamp: "Formatted at 2025-12-14 03:03:29 on dist-test-slave-k7z7"
server_key: "0940147033507b16e9c38adbfc60ba31"
server_key_iv: "7e90e8e376d6015d7e46ab77aa9b1e2b"
server_key_version: "encryptionkey@0"
I20251214 03:03:29.902519 12805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "cb665290b4114b078b3834d543031cd9"
format_stamp: "Formatted at 2025-12-14 03:03:29 on dist-test-slave-k7z7"
server_key: "0940147033507b16e9c38adbfc60ba31"
server_key_iv: "7e90e8e376d6015d7e46ab77aa9b1e2b"
server_key_version: "encryptionkey@0"
I20251214 03:03:29.905640 12805 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:03:29.907686 12820 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:29.908638 12805 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:03:29.908730 12805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "cb665290b4114b078b3834d543031cd9"
format_stamp: "Formatted at 2025-12-14 03:03:29 on dist-test-slave-k7z7"
server_key: "0940147033507b16e9c38adbfc60ba31"
server_key_iv: "7e90e8e376d6015d7e46ab77aa9b1e2b"
server_key_version: "encryptionkey@0"
I20251214 03:03:29.908819 12805 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:29.944432 12805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:29.947329 12805 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:29.947499 12805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:29.948033 12805 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:29.948901 12805 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:29.948949 12805 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:29.948992 12805 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:29.949020 12805 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:29.957569 12805 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:44673
I20251214 03:03:29.957593 12933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:44673 every 8 connection(s)
I20251214 03:03:29.958499 12805 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:03:29.961719 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 12805
I20251214 03:03:29.961835 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:03:29.962086 1655 external_mini_cluster.cc:1468] Setting key 236a3e5a197a513cc3e9a0f1d64a901b
Dec 14 03:03:29 dist-test-slave-k7z7 krb5kdc[12557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
I20251214 03:03:29.970695 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:29.960185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:57933 (local address 127.1.157.254:33113)
1214 03:03:29.960329 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:29.960333 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:29.960990 (+ 657us) server_negotiation.cc:408] Connection header received
1214 03:03:29.961805 (+ 815us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:29.961808 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:29.961866 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:29.961949 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:29.963907 (+ 1958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:29.964384 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:29.965122 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:29.965330 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:29.967310 (+ 1980us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:29.967330 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:29.967333 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:29.967363 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:29.968819 (+ 1456us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:29.969299 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:29.969303 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:29.969306 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:29.969355 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:29.969630 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:29.969634 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:29.969636 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:29.969823 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:29.969927 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:29.970389 (+ 462us) server_negotiation.cc:300] Negotiation successful
1214 03:03:29.970507 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251214 03:03:29.971320 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:29.960455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33113 (local address 127.1.157.194:57933)
1214 03:03:29.960859 (+ 404us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:29.960891 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:29.961615 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:29.962140 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:29.962151 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:29.962645 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:29.963735 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:29.963753 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:29.964498 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:29.964502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:29.965018 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:29.965026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:29.965216 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:29.965750 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:29.965773 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:29.967166 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:29.968929 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:29.968934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:29.968943 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:29.969160 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:29.969439 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:29.969442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:29.969443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:29.969538 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:29.969915 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:29.969920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:29.970137 (+ 217us) client_negotiation.cc:769] Sending connection context
1214 03:03:29.970305 (+ 168us) client_negotiation.cc:240] Negotiation successful
1214 03:03:29.970528 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":103,"threads_started":1}
I20251214 03:03:29.972398 12934 heartbeater.cc:344] Connected to a master server at 127.1.157.254:33113
I20251214 03:03:29.972632 12934 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
I20251214 03:03:29.973148 12934 heartbeater.cc:507] Master 127.1.157.254:33113 requested a full tablet report, sending...
I20251214 03:03:29.974188 12605 ts_manager.cc:194] Registered new tserver with Master: cb665290b4114b078b3834d543031cd9 (127.1.157.194:44673)
I20251214 03:03:29.974949 12605 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:57933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:03:30.006770 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:33113
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:03:30.110906 12941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:03:30.111150 12941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:03:30.111210 12941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:03:30.114502 12941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:03:30.114564 12941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:03:30.114648 12941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:03:30.118734 12941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:03:30.119732 12941 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:03:30.120522 12941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:03:30.126652 12947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:30.126662 12949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:03:30.126814 12946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:03:30.126914 12941 server_base.cc:1047] running on GCE node
I20251214 03:03:30.127223 12941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:03:30.127729 12941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:03:30.128872 12941 hybrid_clock.cc:648] HybridClock initialized: now 1765681410128857 us; error 28 us; skew 500 ppm
Dec 14 03:03:30 dist-test-slave-k7z7 krb5kdc[12557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681410, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:03:30.131502 12941 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:03:30.132450 12941 webserver.cc:492] Webserver started at http://127.1.157.195:32965/ using document root <none> and password file <none>
I20251214 03:03:30.132937 12941 fs_manager.cc:362] Metadata directory not provided
I20251214 03:03:30.132982 12941 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:03:30.133137 12941 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 03:03:30.134769 12941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "38fff6b65d004e48a5601b5c293ed5cc"
format_stamp: "Formatted at 2025-12-14 03:03:30 on dist-test-slave-k7z7"
server_key: "b5264c8fcb977b2057ca3c7430aa6782"
server_key_iv: "6e50ae0add22f703478719cc197ef93b"
server_key_version: "encryptionkey@0"
I20251214 03:03:30.135177 12941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "38fff6b65d004e48a5601b5c293ed5cc"
format_stamp: "Formatted at 2025-12-14 03:03:30 on dist-test-slave-k7z7"
server_key: "b5264c8fcb977b2057ca3c7430aa6782"
server_key_iv: "6e50ae0add22f703478719cc197ef93b"
server_key_version: "encryptionkey@0"
I20251214 03:03:30.138202 12941 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251214 03:03:30.140169 12956 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:30.141100 12941 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251214 03:03:30.141215 12941 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "38fff6b65d004e48a5601b5c293ed5cc"
format_stamp: "Formatted at 2025-12-14 03:03:30 on dist-test-slave-k7z7"
server_key: "b5264c8fcb977b2057ca3c7430aa6782"
server_key_iv: "6e50ae0add22f703478719cc197ef93b"
server_key_version: "encryptionkey@0"
I20251214 03:03:30.141314 12941 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:03:30.162946 12941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:03:30.166158 12941 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:03:30.166373 12941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:03:30.166917 12941 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:03:30.167816 12941 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:03:30.167866 12941 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:30.167918 12941 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:03:30.167943 12941 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:03:30.176609 12941 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:41773
I20251214 03:03:30.176638 13069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:41773 every 8 connection(s)
I20251214 03:03:30.177542 12941 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:03:30.182394 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 12941
I20251214 03:03:30.182521 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:03:30.182775 1655 external_mini_cluster.cc:1468] Setting key 9f0c66a5e1bd510a7de0165e1a804da8
Dec 14 03:03:30 dist-test-slave-k7z7 krb5kdc[12557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681410, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:03:30.189039 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.179261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:53871 (local address 127.1.157.254:33113)
1214 03:03:30.179426 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:30.179431 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:30.180070 (+ 639us) server_negotiation.cc:408] Connection header received
1214 03:03:30.180860 (+ 790us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:30.180863 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:30.180913 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:30.180986 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:30.182850 (+ 1864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.183436 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.184158 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.184336 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.186245 (+ 1909us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:30.186259 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:30.186262 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:30.186285 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:30.187532 (+ 1247us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:30.187955 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:30.187957 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:30.187959 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:30.188003 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:30.188202 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:30.188204 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:30.188205 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:30.188357 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:30.188431 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:30.188833 (+ 402us) server_negotiation.cc:300] Negotiation successful
1214 03:03:30.188926 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251214 03:03:30.189651 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.179545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:33113 (local address 127.1.157.195:53871)
1214 03:03:30.179940 (+ 395us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:30.179975 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:30.180678 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:30.181116 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:30.181123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:30.181625 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:03:30.182696 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.182713 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.183554 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.183557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:30.184053 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.184061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.184224 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.184789 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:03:30.184811 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:03:30.186124 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:03:30.187642 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:30.187646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:30.187654 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:30.187865 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:30.188069 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:03:30.188071 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:03:30.188073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:03:30.188144 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:03:30.188429 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:03:30.188434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:03:30.188636 (+ 202us) client_negotiation.cc:769] Sending connection context
1214 03:03:30.188791 (+ 155us) client_negotiation.cc:240] Negotiation successful
1214 03:03:30.188979 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":86,"threads_started":1}
I20251214 03:03:30.190588 13070 heartbeater.cc:344] Connected to a master server at 127.1.157.254:33113
I20251214 03:03:30.190794 13070 heartbeater.cc:461] Registering TS with master...
I20251214 03:03:30.191249 13070 heartbeater.cc:507] Master 127.1.157.254:33113 requested a full tablet report, sending...
I20251214 03:03:30.192142 12605 ts_manager.cc:194] Registered new tserver with Master: 38fff6b65d004e48a5601b5c293ed5cc (127.1.157.195:41773)
I20251214 03:03:30.192664 12605 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:53871
I20251214 03:03:30.195976 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251214 03:03:30.203348 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.197665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.1.157.254:33113)
1214 03:03:30.197762 (+ 97us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:30.197765 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:30.197908 (+ 143us) server_negotiation.cc:408] Connection header received
1214 03:03:30.198005 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:30.198007 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:30.198042 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:30.198095 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:03:30.198794 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.199257 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.199850 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.199964 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.200887 (+ 923us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:03:30.200899 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:03:30.200901 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:03:30.200914 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:03:30.202215 (+ 1301us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:30.202548 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:30.202550 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:30.202551 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:30.202586 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:03:30.202727 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:03:30.202729 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:03:30.202730 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:03:30.202879 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:03:30.202972 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:30.203128 (+ 156us) server_negotiation.cc:300] Negotiation successful
1214 03:03:30.203210 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251214 03:03:30.205551 12605 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:03:30.207706 12605 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:03:30.219700 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.215623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:44673 (local address 127.0.0.1:33684)
1214 03:03:30.216160 (+ 537us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:30.216198 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:30.216327 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:30.217032 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:30.217037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:30.217056 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:30.217402 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.217410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.218431 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.218434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:30.219224 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.219246 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.219341 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.219396 (+ 55us) client_negotiation.cc:769] Sending connection context
1214 03:03:30.219483 (+ 87us) client_negotiation.cc:240] Negotiation successful
1214 03:03:30.219570 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":81,"threads_started":1}
I20251214 03:03:30.220082 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.215799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:37835 (local address 127.0.0.1:46016)
1214 03:03:30.216754 (+ 955us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:30.216767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:30.216847 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:30.217262 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:30.217266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:30.217284 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:30.217509 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.217518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.218634 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.218638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:30.219531 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.219539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.219633 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.219649 (+ 16us) client_negotiation.cc:769] Sending connection context
1214 03:03:30.219692 (+ 43us) client_negotiation.cc:240] Negotiation successful
1214 03:03:30.219744 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":92,"threads_started":1}
I20251214 03:03:30.220546 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.216258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41773 (local address 127.0.0.1:44620)
1214 03:03:30.216956 (+ 698us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:30.216968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:30.217040 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:30.217780 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:30.217783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:30.217800 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:30.218029 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.218036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.219320 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.219324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:30.220207 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.220217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.220324 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.220340 (+ 16us) client_negotiation.cc:769] Sending connection context
1214 03:03:30.220386 (+ 46us) client_negotiation.cc:240] Negotiation successful
1214 03:03:30.220439 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":78,"threads_started":1}
I20251214 03:03:30.220557 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.216326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46016 (local address 127.1.157.193:37835)
1214 03:03:30.216637 (+ 311us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:30.216643 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:30.216920 (+ 277us) server_negotiation.cc:408] Connection header received
1214 03:03:30.216984 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:30.216987 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:30.217102 (+ 115us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:30.217240 (+ 138us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:30.217635 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.218527 (+ 892us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.219658 (+ 1131us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.220161 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.220270 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:30.220331 (+ 61us) server_negotiation.cc:300] Negotiation successful
1214 03:03:30.220414 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":59,"threads_started":1}
I20251214 03:03:30.220880 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.215734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.1.157.194:44673)
1214 03:03:30.216074 (+ 340us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:30.216080 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:30.216453 (+ 373us) server_negotiation.cc:408] Connection header received
1214 03:03:30.216554 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:30.216559 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:30.216735 (+ 176us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:30.216866 (+ 131us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:30.217533 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.218329 (+ 796us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.220024 (+ 1695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.220539 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.220644 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:30.220703 (+ 59us) server_negotiation.cc:300] Negotiation successful
1214 03:03:30.220778 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":86,"threads_started":1}
I20251214 03:03:30.222086 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.216610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.1.157.195:41773)
1214 03:03:30.217289 (+ 679us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:30.217295 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:30.217314 (+ 19us) server_negotiation.cc:408] Connection header received
1214 03:03:30.217383 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:30.217387 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:30.217534 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:30.217647 (+ 113us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:30.218152 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.219184 (+ 1032us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.220453 (+ 1269us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.221736 (+ 1283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.221822 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:30.221881 (+ 59us) server_negotiation.cc:300] Negotiation successful
1214 03:03:30.221958 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":133,"threads_started":1}
I20251214 03:03:30.223328 12732 tablet_service.cc:1505] Processing CreateTablet for tablet 421c881a034f4f35835495d53b13b154 (DEFAULT_TABLE table=test-table [id=e8bb2f06859a40c98dfbdbb3dcb203cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:30.223435 12868 tablet_service.cc:1505] Processing CreateTablet for tablet 421c881a034f4f35835495d53b13b154 (DEFAULT_TABLE table=test-table [id=e8bb2f06859a40c98dfbdbb3dcb203cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:30.224128 12732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421c881a034f4f35835495d53b13b154. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:30.224288 12868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421c881a034f4f35835495d53b13b154. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:30.224342 13004 tablet_service.cc:1505] Processing CreateTablet for tablet 421c881a034f4f35835495d53b13b154 (DEFAULT_TABLE table=test-table [id=e8bb2f06859a40c98dfbdbb3dcb203cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:03:30.225083 13004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421c881a034f4f35835495d53b13b154. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:03:30.229225 13087 tablet_bootstrap.cc:492] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a: Bootstrap starting.
I20251214 03:03:30.230435 13088 tablet_bootstrap.cc:492] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9: Bootstrap starting.
I20251214 03:03:30.231428 13089 tablet_bootstrap.cc:492] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc: Bootstrap starting.
I20251214 03:03:30.231856 13087 tablet_bootstrap.cc:654] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:30.232494 13088 tablet_bootstrap.cc:654] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:30.232668 13087 log.cc:826] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:30.232965 13089 tablet_bootstrap.cc:654] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc: Neither blocks nor log segments found. Creating new log.
I20251214 03:03:30.233310 13088 log.cc:826] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:30.233618 13089 log.cc:826] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc: Log is configured to *not* fsync() on all Append() calls
I20251214 03:03:30.234468 13087 tablet_bootstrap.cc:492] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a: No bootstrap required, opened a new log
I20251214 03:03:30.234666 13087 ts_tablet_manager.cc:1403] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251214 03:03:30.234985 13088 tablet_bootstrap.cc:492] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9: No bootstrap required, opened a new log
I20251214 03:03:30.235158 13088 ts_tablet_manager.cc:1403] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251214 03:03:30.235677 13089 tablet_bootstrap.cc:492] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc: No bootstrap required, opened a new log
I20251214 03:03:30.235829 13089 ts_tablet_manager.cc:1403] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251214 03:03:30.237782 13087 raft_consensus.cc:359] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb665290b4114b078b3834d543031cd9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 44673 } } peers { permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 } } peers { permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 } }
I20251214 03:03:30.237993 13087 raft_consensus.cc:385] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:30.238046 13087 raft_consensus.cc:740] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb463dd87654688bdfac813d4a1c98a, State: Initialized, Role: FOLLOWER
I20251214 03:03:30.238471 13087 consensus_queue.cc:260] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb665290b4114b078b3834d543031cd9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 44673 } } peers { permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 } } peers { permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 } }
I20251214 03:03:30.238559 13088 raft_consensus.cc:359] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb665290b4114b078b3834d543031cd9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 44673 } } peers { permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 } } peers { permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 } }
I20251214 03:03:30.238742 13088 raft_consensus.cc:385] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:30.238803 13088 raft_consensus.cc:740] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb665290b4114b078b3834d543031cd9, State: Initialized, Role: FOLLOWER
I20251214 03:03:30.238871 13089 raft_consensus.cc:359] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb665290b4114b078b3834d543031cd9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 44673 } } peers { permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 } } peers { permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 } }
I20251214 03:03:30.239039 13089 raft_consensus.cc:385] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:03:30.239096 13089 raft_consensus.cc:740] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fff6b65d004e48a5601b5c293ed5cc, State: Initialized, Role: FOLLOWER
I20251214 03:03:30.239189 13088 consensus_queue.cc:260] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb665290b4114b078b3834d543031cd9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 44673 } } peers { permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 } } peers { permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 } }
I20251214 03:03:30.239286 13087 ts_tablet_manager.cc:1434] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251214 03:03:30.239460 13089 consensus_queue.cc:260] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb665290b4114b078b3834d543031cd9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 44673 } } peers { permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 } } peers { permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 } }
I20251214 03:03:30.239632 12798 heartbeater.cc:499] Master 127.1.157.254:33113 was elected leader, sending a full tablet report...
I20251214 03:03:30.239869 12934 heartbeater.cc:499] Master 127.1.157.254:33113 was elected leader, sending a full tablet report...
I20251214 03:03:30.240123 13089 ts_tablet_manager.cc:1434] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20251214 03:03:30.240641 13070 heartbeater.cc:499] Master 127.1.157.254:33113 was elected leader, sending a full tablet report...
I20251214 03:03:30.240801 13088 ts_tablet_manager.cc:1434] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20251214 03:03:30.315562 13094 raft_consensus.cc:493] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:03:30.315748 13094 raft_consensus.cc:515] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb665290b4114b078b3834d543031cd9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 44673 } } peers { permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 } } peers { permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 } }
I20251214 03:03:30.316839 13094 leader_election.cc:290] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eb463dd87654688bdfac813d4a1c98a (127.1.157.193:37835), 38fff6b65d004e48a5601b5c293ed5cc (127.1.157.195:41773)
I20251214 03:03:30.320377 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.317314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:41773 (local address 127.1.157.194:56191)
1214 03:03:30.317725 (+ 411us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:30.317735 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:30.317820 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:30.318148 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:30.318152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:30.318169 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:30.318466 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.318473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.319391 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.319395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:30.319997 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.320004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.320105 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.320127 (+ 22us) client_negotiation.cc:769] Sending connection context
1214 03:03:30.320174 (+ 47us) client_negotiation.cc:240] Negotiation successful
1214 03:03:30.320230 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":154,"threads_started":1}
I20251214 03:03:30.320624 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.317054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:37835 (local address 127.1.157.194:36735)
1214 03:03:30.317163 (+ 109us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:30.317197 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:30.317297 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:30.317688 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:03:30.317691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:03:30.317712 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:03:30.317984 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.317992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.319311 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.319315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:03:30.320254 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:03:30.320264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.320376 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.320395 (+ 19us) client_negotiation.cc:769] Sending connection context
1214 03:03:30.320438 (+ 43us) client_negotiation.cc:240] Negotiation successful
1214 03:03:30.320505 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20251214 03:03:30.320832 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.317536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:56191 (local address 127.1.157.195:41773)
1214 03:03:30.317666 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:30.317670 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:30.317800 (+ 130us) server_negotiation.cc:408] Connection header received
1214 03:03:30.317982 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:30.317989 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:30.318053 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:30.318143 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:30.318585 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.319292 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.320106 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.320567 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.320613 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:30.320660 (+ 47us) server_negotiation.cc:300] Negotiation successful
1214 03:03:30.320723 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251214 03:03:30.321460 13024 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "421c881a034f4f35835495d53b13b154" candidate_uuid: "cb665290b4114b078b3834d543031cd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38fff6b65d004e48a5601b5c293ed5cc" is_pre_election: true
I20251214 03:03:30.321537 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.317156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:36735 (local address 127.1.157.193:37835)
1214 03:03:30.317341 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:30.317346 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:30.317363 (+ 17us) server_negotiation.cc:408] Connection header received
1214 03:03:30.317419 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:30.317422 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:30.317473 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:30.317565 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:03:30.318133 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.319192 (+ 1059us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.320881 (+ 1689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.321325 (+ 444us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.321356 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:30.321398 (+ 42us) server_negotiation.cc:300] Negotiation successful
1214 03:03:30.321442 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20251214 03:03:30.321753 13024 raft_consensus.cc:2468] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb665290b4114b078b3834d543031cd9 in term 0.
I20251214 03:03:30.322021 12752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "421c881a034f4f35835495d53b13b154" candidate_uuid: "cb665290b4114b078b3834d543031cd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb463dd87654688bdfac813d4a1c98a" is_pre_election: true
I20251214 03:03:30.322257 12752 raft_consensus.cc:2468] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb665290b4114b078b3834d543031cd9 in term 0.
I20251214 03:03:30.322399 12822 leader_election.cc:304] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38fff6b65d004e48a5601b5c293ed5cc, cb665290b4114b078b3834d543031cd9; no voters:
I20251214 03:03:30.322655 13094 raft_consensus.cc:2804] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:03:30.322721 13094 raft_consensus.cc:493] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:03:30.322767 13094 raft_consensus.cc:3060] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:30.323623 13094 raft_consensus.cc:515] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb665290b4114b078b3834d543031cd9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 44673 } } peers { permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 } } peers { permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 } }
I20251214 03:03:30.323944 13094 leader_election.cc:290] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [CANDIDATE]: Term 1 election: Requested vote from peers 0eb463dd87654688bdfac813d4a1c98a (127.1.157.193:37835), 38fff6b65d004e48a5601b5c293ed5cc (127.1.157.195:41773)
I20251214 03:03:30.324298 13024 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "421c881a034f4f35835495d53b13b154" candidate_uuid: "cb665290b4114b078b3834d543031cd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38fff6b65d004e48a5601b5c293ed5cc"
I20251214 03:03:30.324301 12752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "421c881a034f4f35835495d53b13b154" candidate_uuid: "cb665290b4114b078b3834d543031cd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb463dd87654688bdfac813d4a1c98a"
I20251214 03:03:30.324407 13024 raft_consensus.cc:3060] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:30.324414 12752 raft_consensus.cc:3060] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:03:30.325302 13024 raft_consensus.cc:2468] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb665290b4114b078b3834d543031cd9 in term 1.
I20251214 03:03:30.325580 12822 leader_election.cc:304] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38fff6b65d004e48a5601b5c293ed5cc, cb665290b4114b078b3834d543031cd9; no voters:
I20251214 03:03:30.325649 12752 raft_consensus.cc:2468] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb665290b4114b078b3834d543031cd9 in term 1.
I20251214 03:03:30.325721 13094 raft_consensus.cc:2804] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:03:30.325911 13094 raft_consensus.cc:697] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [term 1 LEADER]: Becoming Leader. State: Replica: cb665290b4114b078b3834d543031cd9, State: Running, Role: LEADER
I20251214 03:03:30.326193 13094 consensus_queue.cc:237] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb665290b4114b078b3834d543031cd9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 44673 } } peers { permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 } } peers { permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 } }
I20251214 03:03:30.329013 12604 catalog_manager.cc:5654] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb665290b4114b078b3834d543031cd9 (127.1.157.194). New cstate: current_term: 1 leader_uuid: "cb665290b4114b078b3834d543031cd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb665290b4114b078b3834d543031cd9" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 } health_report { overall_health: UNKNOWN } } }
I20251214 03:03:30.340759 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.337890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.1.157.194:44673)
1214 03:03:30.338027 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1214 03:03:30.338031 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:03:30.338044 (+ 13us) server_negotiation.cc:408] Connection header received
1214 03:03:30.338113 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:03:30.338115 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:03:30.338167 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:03:30.338274 (+ 107us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:03:30.338595 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.339086 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:03:30.339708 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:03:30.339878 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:03:30.339964 (+ 86us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:03:30.340401 (+ 437us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:03:30.340503 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1214 03:03:30.340578 (+ 75us) server_negotiation.cc:300] Negotiation successful
1214 03:03:30.340634 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251214 03:03:30.346010 12752 raft_consensus.cc:1275] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a [term 1 FOLLOWER]: Refusing update from remote peer cb665290b4114b078b3834d543031cd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:30.346010 13024 raft_consensus.cc:1275] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc [term 1 FOLLOWER]: Refusing update from remote peer cb665290b4114b078b3834d543031cd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:03:30.346660 13094 consensus_queue.cc:1048] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eb463dd87654688bdfac813d4a1c98a" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:30.346836 13097 consensus_queue.cc:1048] T 421c881a034f4f35835495d53b13b154 P cb665290b4114b078b3834d543031cd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38fff6b65d004e48a5601b5c293ed5cc" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 41773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:03:30.355608 13101 mvcc.cc:204] Tried to move back new op lower bound from 7232231056770084864 to 7232231056698531840. Current Snapshot: MvccSnapshot[applied={T|T < 7232231056770084864}]
I20251214 03:03:30.357100 13100 mvcc.cc:204] Tried to move back new op lower bound from 7232231056770084864 to 7232231056698531840. Current Snapshot: MvccSnapshot[applied={T|T < 7232231056770084864}]
I20251214 03:03:30.364689 12604 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59128:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:30.364835 12604 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59128:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:03:30.367096 12604 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ab08b7d88cb24738ab9ca2371b75e906: Sending DeleteTablet for 3 replicas of tablet 421c881a034f4f35835495d53b13b154
I20251214 03:03:30.367723 12868 tablet_service.cc:1552] Processing DeleteTablet for tablet 421c881a034f4f35835495d53b13b154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:30 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:33684
I20251214 03:03:30.367909 13004 tablet_service.cc:1552] Processing DeleteTablet for tablet 421c881a034f4f35835495d53b13b154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:30 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:44620
I20251214 03:03:30.368448 12732 tablet_service.cc:1552] Processing DeleteTablet for tablet 421c881a034f4f35835495d53b13b154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:03:30 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:46016
I20251214 03:03:30.368892 13112 tablet_replica.cc:333] T 421c881a034f4f35835495d53b13b154 P 0eb463dd87654688bdfac813d4a1c98a: stopping tablet replica
I20251214 03:03:30.368994 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 12669
W20251214 03:03:30.374827 12590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.157.193:37835 (error 108)
I20251214 03:03:30.375092 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 12805
I20251214 03:03:30.375949 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:03:30.375637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:37835 (local address 127.0.0.1:46018)
1214 03:03:30.375814 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:30.375867 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.193:37835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20251214 03:03:30.376085 12590 catalog_manager.cc:4712] TS 0eb463dd87654688bdfac813d4a1c98a (127.1.157.193:37835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 421c881a034f4f35835495d53b13b154: Network error: Client connection negotiation failed: client connection to 127.1.157.193:37835: connect: Connection refused (error 111)
I20251214 03:03:30.379055 13111 tablet_replica.cc:333] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc: stopping tablet replica
I20251214 03:03:30.379347 13111 raft_consensus.cc:2243] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:03:30.379710 13111 raft_consensus.cc:2272] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:03:30.381263 13111 ts_tablet_manager.cc:1916] T 421c881a034f4f35835495d53b13b154 P 38fff6b65d004e48a5601b5c293ed5cc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 03:03:30.381753 13083 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:03:30.381201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:44673 (local address 127.0.0.1:33702)
1214 03:03:30.381363 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1214 03:03:30.381378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:03:30.381461 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:03:30.381661 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.194:44673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":90}
I20251214 03:03:30.382231 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 12941
I20251214 03:03:30.386900 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 12573
2025-12-14T03:03:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3125 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:03:30 dist-test-slave-k7z7 krb5kdc[13116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:03:30 dist-test-slave-k7z7 krb5kdc[13116](info): set up 2 sockets
Dec 14 03:03:30 dist-test-slave-k7z7 krb5kdc[13116](info): commencing operation
krb5kdc: starting...
W20251214 03:03:32.221698 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.814s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:03:32 dist-test-slave-k7z7 krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:03:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:03:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/rangeradmin_127.1.157.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/rangeradmin_127.1.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/rangerlookup_127.1.157.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/rangerlookup_127.1.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/HTTP_127.1.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/HTTP_127.1.157.212@KRBTEST.COM.keytab.
I20251214 03:03:32.365533 1655 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_d3z9H/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/postgres -l logfile start
2025-12-14 03:03:34.917 UTC [13147] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-12-14 03:03:34.917 UTC [13147] LOG: listening on IPv4 address "127.1.157.212", port 50221
2025-12-14 03:03:34.921 UTC [13147] LOG: listening on Unix socket "/tmp/.s.PGSQL.50221"
2025-12-14 03:03:34.928 UTC [13152] LOG: database system was shut down at 2025-12-14 03:03:33 UTC
2025-12-14 03:03:34.934 UTC [13147] LOG: database system is ready to accept connections
I20251214 03:03:36.693372 1655 mini_postgres.cc:96] Postgres bound to 50221
2025-12-14 03:03:36.698 UTC [13158] FATAL: database "slave" does not exist
127.1.157.212:50221 - accepting connections
I20251214 03:03:36.699533 1655 mini_ranger.cc:161] Starting Ranger...
I20251214 03:03:36.717939 1655 mini_ranger.cc:84] Created miniranger Postgres user
I20251214 03:03:36.783039 1655 mini_ranger.cc:87] Created ranger Postgres database
I20251214 03:03:36.783156 1655 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-admin
2025-12-14 03:03:37,063 [I] DB FLAVOR :POSTGRES
2025-12-14 03:03:37,063 [I] --------- Verifying Ranger DB connection ---------
2025-12-14 03:03:37,063 [I] Checking connection..
2025-12-14 03:03:37,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-12-14 03:03:37,391 [I] Checking connection passed.
2025-12-14 03:03:37,392 [I] --------- Verifying version history table ---------
2025-12-14 03:03:37,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-14 03:03:37,705 [I] Table x_db_version_h does not exist in database ranger
2025-12-14 03:03:37,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-14 03:03:38,040 [I] Table x_db_version_h does not exist in database ranger
2025-12-14 03:03:38,040 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-12-14 03:03:38,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-12-14 03:03:38.348 UTC [13225] WARNING: there is no transaction in progress
2025-12-14 03:03:38,353 [I] create_dbversion_catalog.sql file imported successfully
2025-12-14 03:03:38,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-14 03:03:38,686 [I] Table x_db_version_h already exists in database 'ranger'
2025-12-14 03:03:38,686 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-14 03:03:38,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-12-14 03:03:39,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-12-14 03:03:39,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k7z7.c.gcp-upstream.internal','N') ;"
2025-12-14 03:03:39,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-14 03:03:39,979 [I] Table x_portal_user does not exist in database ranger
2025-12-14 03:03:39,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-14 03:03:40,299 [I] Table x_policy_ref_group does not exist in database ranger
2025-12-14 03:03:40,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-14 03:03:40,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-14 03:03:40,930 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-12-14 03:03:40,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-12-14 03:03:41.757 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.768 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.781 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.793 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.804 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.817 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.867 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.873 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.883 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.893 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.903 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.915 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.924 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.928 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.932 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:41.935 UTC [13342] WARNING: there is no transaction in progress
2025-12-14 03:03:42,383 [I] ranger_core_db_postgres.sql file imported successfully
2025-12-14 03:03:42,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-12-14 03:03:42,689 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-12-14 03:03:42,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-14 03:03:43,007 [I] Table x_portal_user already exists in database 'ranger'
2025-12-14 03:03:43,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-14 03:03:43,332 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-12-14 03:03:43,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-14 03:03:43,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-14 03:03:43,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k7z7.c.gcp-upstream.internal';"
2025-12-14 03:03:44,281 [I] CORE_DB_SCHEMA import status has been updated
2025-12-14 03:03:44,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-12-14 03:03:44,584 [I] DB_PATCHES have already been applied
I20251214 03:03:44.590133 1655 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-admin:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_d3z9H/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251214 03:03:44.590201 1655 mini_ranger.cc:193] Using host: 127.1.157.212
I20251214 03:03:44.593138 1655 mini_ranger.cc:227] Ranger admin URL: http://127.1.157.212:45955
Dec 14, 2025 3:03:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 14, 2025 3:03:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 14, 2025 3:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-admin/ews/webapp
Dec 14, 2025 3:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Dec 14, 2025 3:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-admin/ews/webapp] .....
Dec 14, 2025 3:03:45 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Dec 14, 2025 3:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-admin/ews/webapp].
Dec 14, 2025 3:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Dec 14, 2025 3:03:45 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-45955"]
Dec 14, 2025 3:03:45 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Dec 14, 2025 3:03:45 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Dec 14, 2025 3:03:45 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Dec 14, 2025 3:03:45 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Dec 14, 2025 3:03:45 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Dec 14, 2025 3:03:46 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Dec 14, 2025 3:03:46 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 14, 2025 3:03:46 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Dec 14, 2025 3:03:46 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20251214 03:03:46.612885 1655 mini_ranger.cc:160] Time spent starting Ranger: real 9.913s user 0.000s sys 0.009s
Dec 14, 2025 3:04:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Dec 14, 2025 3:04:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
Dec 14, 2025 3:04:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Dec 14, 2025 3:04:04 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Dec 14, 2025 3:04:05 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Dec 14, 2025 3:04:05 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-45955"]
I20251214 03:04:06.107362 1655 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/rangerkms_127.1.157.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/rangerkms_127.1.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.157.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.157.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/HTTP_127.1.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.157.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/HTTP_127.1.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Dec 14 03:04:06 dist-test-slave-k7z7 krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681446, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251214 03:04:06.187932 1655 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251214 03:04:06.197360 1655 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251214 03:04:06.293959 1655 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251214 03:04:06.294044 1655 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-kms
I20251214 03:04:06.294068 1655 mini_ranger_kms.cc:227] Using postgres at 127.1.157.212:50221
2025-12-14 03:04:06,446 [I] DB FLAVOR :POSTGRES
2025-12-14 03:04:06,446 [I] --------- Verifying Ranger DB connection ---------
2025-12-14 03:04:06,446 [I] Checking connection
2025-12-14 03:04:06,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-12-14 03:04:06,764 [I] connection success
2025-12-14 03:04:06,764 [I] --------- Verifying Ranger DB tables ---------
2025-12-14 03:04:06,765 [I] Verifying table ranger_masterkey in database rangerkms
2025-12-14 03:04:06,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-12-14 03:04:07,079 [I] Table ranger_masterkey does not exist in database rangerkms
2025-12-14 03:04:07,079 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-14 03:04:07,079 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-12-14 03:04:07,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:50221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-12-14 03:04:07,400 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251214 03:04:07.824311 1655 mini_ranger_kms.cc:310] Created kms service
I20251214 03:04:08.030628 1655 mini_ranger_kms.cc:326] Created kudu user
I20251214 03:04:08.070142 1655 mini_ranger_kms.cc:343] Created rangerkms user
Dec 14 03:04:08 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681446, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
I20251214 03:04:08.462656 1655 mini_ranger_kms.cc:384] Added ranger policy
I20251214 03:04:08.462827 1655 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-kms:/tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251214 03:04:08.462847 1655 mini_ranger_kms.cc:240] Using host: 127.1.157.212
I20251214 03:04:08.466470 1655 mini_ranger_kms.cc:276] Ranger KMS PID: 13534
I20251214 03:04:08.466563 1655 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.157.212:39389
Dec 14, 2025 3:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 14, 2025 3:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 14, 2025 3:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Dec 14, 2025 3:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-kms/ews/webapp] .....
Dec 14, 2025 3:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-kms/ews/webapp].
Dec 14, 2025 3:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Dec 14, 2025 3:04:09 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-39389"]
Dec 14, 2025 3:04:09 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Dec 14, 2025 3:04:09 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Dec 14, 2025 3:04:10 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Dec 14, 2025 3:04:10 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 14 03:04:10 dist-test-slave-k7z7 krb5kdc[13116](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1765681450, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.157.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:04:10.461755 1655 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.274s user 0.016s sys 0.118s
I20251214 03:04:10.461887 1655 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251214 03:04:10.461966 1655 mini_ranger_kms.cc:401] 127.1.157.212:39389/kms/v1/keys
Dec 14 03:04:10 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681450, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.157.212@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
Dec 14 03:04:11 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681450, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.157.212@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
Dec 14 03:04:13 dist-test-slave-k7z7 krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:04:13.438467 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:46221
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:46221
--ranger_config_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.157.212:39389/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:04:13.544147 13587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:04:13.544366 13587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:04:13.544435 13587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:04:13.547646 13587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:04:13.547703 13587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:04:13.547732 13587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:04:13.547765 13587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:04:13.547796 13587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:04:13.551954 13587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.157.212:39389/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:46221
--ranger_config_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:46221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:04:13.553040 13587 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:04:13.553866 13587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:04:13.559425 13595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:04:13.559425 13593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:04:13.559552 13587 server_base.cc:1047] running on GCE node
W20251214 03:04:13.559688 13592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:04:13.560158 13587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:04:13.560992 13587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:04:13.562141 13587 hybrid_clock.cc:648] HybridClock initialized: now 1765681453562124 us; error 30 us; skew 500 ppm
Dec 14 03:04:13 dist-test-slave-k7z7 krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:04:13.564471 13587 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:04:13.565536 13587 webserver.cc:492] Webserver started at http://127.1.157.254:37795/ using document root <none> and password file <none>
I20251214 03:04:13.566045 13587 fs_manager.cc:362] Metadata directory not provided
I20251214 03:04:13.566092 13587 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:04:13.566257 13587 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 14 03:04:13 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
I20251214 03:04:13.629595 13587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "f90e6242c6284c7aae679482a5e9ff1c"
format_stamp: "Formatted at 2025-12-14 03:04:13 on dist-test-slave-k7z7"
server_key: "d9a05f7748be72f7001f6632db8ec5bf"
server_key_iv: "4278a453982d79729941b0bbf82804bc"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:13.630311 13587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "f90e6242c6284c7aae679482a5e9ff1c"
format_stamp: "Formatted at 2025-12-14 03:04:13 on dist-test-slave-k7z7"
server_key: "d9a05f7748be72f7001f6632db8ec5bf"
server_key_iv: "4278a453982d79729941b0bbf82804bc"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:13.634987 13587 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251214 03:04:13.648847 13602 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:04:13.649930 13587 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 03:04:13.650048 13587 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "f90e6242c6284c7aae679482a5e9ff1c"
format_stamp: "Formatted at 2025-12-14 03:04:13 on dist-test-slave-k7z7"
server_key: "d9a05f7748be72f7001f6632db8ec5bf"
server_key_iv: "4278a453982d79729941b0bbf82804bc"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:13.650157 13587 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:04:13.670521 13587 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:04:13.673393 13587 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:04:13.673568 13587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:04:13.681345 13587 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:46221
I20251214 03:04:13.681352 13654 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:46221 every 8 connection(s)
I20251214 03:04:13.682345 13587 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:04:13.685001 13655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:04:13.685271 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 13587
I20251214 03:04:13.685407 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:04:13.685705 1655 external_mini_cluster.cc:1468] Setting key f38a755d629458dd2a354c18f1a4ef95
I20251214 03:04:13.691139 13655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c: Bootstrap starting.
I20251214 03:04:13.693670 13655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c: Neither blocks nor log segments found. Creating new log.
Dec 14 03:04:13 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:04:13.694557 13655 log.cc:826] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c: Log is configured to *not* fsync() on all Append() calls
I20251214 03:04:13.696815 13655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c: No bootstrap required, opened a new log
I20251214 03:04:13.700176 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:13.687061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.1.157.254:46221)
1214 03:04:13.687532 (+ 471us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:13.687543 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:13.687582 (+ 39us) server_negotiation.cc:408] Connection header received
1214 03:04:13.688224 (+ 642us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:13.688248 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:13.688621 (+ 373us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:13.688954 (+ 333us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:04:13.690169 (+ 1215us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:13.691298 (+ 1129us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:13.692169 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:13.692461 (+ 292us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:13.695226 (+ 2765us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:04:13.695245 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:04:13.695256 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:04:13.695290 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:04:13.697373 (+ 2083us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:04:13.697878 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:04:13.697885 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:04:13.697891 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:04:13.697975 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:04:13.698225 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:04:13.698228 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:04:13.698230 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:04:13.698695 (+ 465us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:04:13.698925 (+ 230us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:13.699171 (+ 246us) server_negotiation.cc:300] Negotiation successful
1214 03:04:13.699402 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":151,"threads_started":1}
I20251214 03:04:13.700109 13655 raft_consensus.cc:359] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90e6242c6284c7aae679482a5e9ff1c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46221 } }
I20251214 03:04:13.700299 13655 raft_consensus.cc:385] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:04:13.700357 13655 raft_consensus.cc:740] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90e6242c6284c7aae679482a5e9ff1c, State: Initialized, Role: FOLLOWER
I20251214 03:04:13.700841 13655 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90e6242c6284c7aae679482a5e9ff1c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46221 } }
I20251214 03:04:13.700968 13655 raft_consensus.cc:399] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:04:13.701041 13655 raft_consensus.cc:493] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:04:13.701131 13655 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:04:13.701959 13655 raft_consensus.cc:515] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90e6242c6284c7aae679482a5e9ff1c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46221 } }
I20251214 03:04:13.702318 13655 leader_election.cc:304] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f90e6242c6284c7aae679482a5e9ff1c; no voters:
I20251214 03:04:13.702639 13655 leader_election.cc:290] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:04:13.702746 13660 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:04:13.702944 13660 raft_consensus.cc:697] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [term 1 LEADER]: Becoming Leader. State: Replica: f90e6242c6284c7aae679482a5e9ff1c, State: Running, Role: LEADER
I20251214 03:04:13.703307 13660 consensus_queue.cc:237] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90e6242c6284c7aae679482a5e9ff1c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46221 } }
I20251214 03:04:13.703742 13655 sys_catalog.cc:565] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:04:13.704547 13660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90e6242c6284c7aae679482a5e9ff1c. Latest consensus state: current_term: 1 leader_uuid: "f90e6242c6284c7aae679482a5e9ff1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90e6242c6284c7aae679482a5e9ff1c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46221 } } }
I20251214 03:04:13.704676 13660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [sys.catalog]: This master's current role is: LEADER
I20251214 03:04:13.705039 13669 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:04:13.705873 13661 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f90e6242c6284c7aae679482a5e9ff1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90e6242c6284c7aae679482a5e9ff1c" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 46221 } } }
I20251214 03:04:13.705955 13661 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c [sys.catalog]: This master's current role is: LEADER
I20251214 03:04:13.708238 13669 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:04:13.708644 13655 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251214 03:04:13.719329 13669 catalog_manager.cc:1357] Generated new cluster ID: 9206262b60dd4dadbbe816480d91aee3
I20251214 03:04:13.719421 13669 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:04:13.770624 13669 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:04:13.772162 13669 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:04:13.779625 13669 catalog_manager.cc:6027] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c: Generated new TSK 0
I20251214 03:04:13.780658 13669 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:04:14.576011 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.157.212:39389/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:46221
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:04:14.698539 13701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:04:14.698848 13701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:04:14.698930 13701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:04:14.703748 13701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:04:14.703845 13701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:04:14.703969 13701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:04:14.708943 13701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.157.212:39389/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:04:14.710415 13701 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:04:14.711551 13701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:04:14.719884 13711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:04:14.720729 13709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:04:14.721659 13701 server_base.cc:1047] running on GCE node
W20251214 03:04:14.723330 13708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:04:14.723868 13701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:04:14.724562 13701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:04:14.727898 13701 hybrid_clock.cc:648] HybridClock initialized: now 1765681454727864 us; error 60 us; skew 500 ppm
Dec 14 03:04:14 dist-test-slave-k7z7 krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:04:14.731293 13701 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:04:14.732686 13701 webserver.cc:492] Webserver started at http://127.1.157.193:35451/ using document root <none> and password file <none>
I20251214 03:04:14.733448 13701 fs_manager.cc:362] Metadata directory not provided
I20251214 03:04:14.733564 13701 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:04:14.733819 13701 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 14 03:04:14 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
I20251214 03:04:14.749814 13701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "723771a6707f49339f986cf02d934472"
format_stamp: "Formatted at 2025-12-14 03:04:14 on dist-test-slave-k7z7"
server_key: "c2a65de7ff9da4f801d1e73a1a490a33"
server_key_iv: "f2da5e7de5cf8420d6b6b05e7d7c7ef4"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:14.750531 13701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "723771a6707f49339f986cf02d934472"
format_stamp: "Formatted at 2025-12-14 03:04:14 on dist-test-slave-k7z7"
server_key: "c2a65de7ff9da4f801d1e73a1a490a33"
server_key_iv: "f2da5e7de5cf8420d6b6b05e7d7c7ef4"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:14.758500 13701 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.000s
I20251214 03:04:14.774026 13720 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:04:14.775313 13701 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251214 03:04:14.775424 13701 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "723771a6707f49339f986cf02d934472"
format_stamp: "Formatted at 2025-12-14 03:04:14 on dist-test-slave-k7z7"
server_key: "c2a65de7ff9da4f801d1e73a1a490a33"
server_key_iv: "f2da5e7de5cf8420d6b6b05e7d7c7ef4"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:14.775528 13701 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:04:14.791514 13701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:04:14.798552 13701 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:04:14.798820 13701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:04:14.799638 13701 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:04:14.800757 13701 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:04:14.800812 13701 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:04:14.800865 13701 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:04:14.800906 13701 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:04:14.811635 13701 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:39101
I20251214 03:04:14.812004 13833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:39101 every 8 connection(s)
I20251214 03:04:14.812777 13701 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:04:14.813989 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 13701
I20251214 03:04:14.814114 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:04:14.814390 1655 external_mini_cluster.cc:1468] Setting key e88c77cdd5b78ed22bfbcd1030632019
Dec 14 03:04:14 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
I20251214 03:04:14.832141 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:14.816000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:34335 (local address 127.1.157.254:46221)
1214 03:04:14.817847 (+ 1847us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:14.817852 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:14.817866 (+ 14us) server_negotiation.cc:408] Connection header received
1214 03:04:14.817909 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:14.817912 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:14.817963 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:14.818042 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:04:14.819685 (+ 1643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:14.820444 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:14.822161 (+ 1717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:14.822328 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:14.826413 (+ 4085us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:04:14.826440 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:04:14.826443 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:04:14.826473 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:04:14.828013 (+ 1540us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:04:14.828823 (+ 810us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:04:14.828827 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:04:14.828829 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:04:14.828898 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:04:14.829385 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:04:14.829389 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:04:14.829392 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:04:14.829614 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:04:14.829697 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:14.830508 (+ 811us) server_negotiation.cc:300] Negotiation successful
1214 03:04:14.831944 (+ 1436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1759,"thread_start_us":79,"threads_started":1}
I20251214 03:04:14.833676 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:14.815039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:46221 (local address 127.1.157.193:34335)
1214 03:04:14.816422 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
1214 03:04:14.816473 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1214 03:04:14.817406 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:04:14.818353 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:04:14.818365 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:04:14.818703 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:04:14.819486 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:14.819505 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:14.820598 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:04:14.820602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:04:14.821152 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:14.821164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:14.821459 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:14.824337 (+ 2878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:04:14.824416 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:04:14.826197 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:04:14.828157 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:04:14.828165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:04:14.828180 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:04:14.828526 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:04:14.829037 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:04:14.829041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:04:14.829043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:04:14.829230 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:04:14.829789 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:04:14.829797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:04:14.830153 (+ 356us) client_negotiation.cc:769] Sending connection context
1214 03:04:14.832465 (+ 2312us) client_negotiation.cc:240] Negotiation successful
1214 03:04:14.832784 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"thread_start_us":116,"threads_started":1}
I20251214 03:04:14.835309 13834 heartbeater.cc:344] Connected to a master server at 127.1.157.254:46221
I20251214 03:04:14.835577 13834 heartbeater.cc:461] Registering TS with master...
I20251214 03:04:14.836127 13834 heartbeater.cc:507] Master 127.1.157.254:46221 requested a full tablet report, sending...
I20251214 03:04:14.837909 13619 ts_manager.cc:194] Registered new tserver with Master: 723771a6707f49339f986cf02d934472 (127.1.157.193:39101)
I20251214 03:04:14.839562 13619 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:34335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Dec 14 03:04:14 dist-test-slave-k7z7 krb5kdc[13116](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1765681454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:04:14.874150 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.157.212:39389/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:46221
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:04:15.029405 13842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:04:15.029755 13842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:04:15.029839 13842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:04:15.034644 13842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:04:15.034734 13842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:04:15.034852 13842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:04:15.040692 13842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.157.212:39389/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:04:15.042050 13842 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:04:15.043110 13842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:04:15.050980 13848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:04:15.051095 13849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:04:15.051303 13851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:04:15.051899 13842 server_base.cc:1047] running on GCE node
I20251214 03:04:15.052317 13842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:04:15.052943 13842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:04:15.054119 13842 hybrid_clock.cc:648] HybridClock initialized: now 1765681455054100 us; error 38 us; skew 500 ppm
Dec 14 03:04:15 dist-test-slave-k7z7 krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:04:15.057484 13842 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:04:15.058786 13842 webserver.cc:492] Webserver started at http://127.1.157.194:44507/ using document root <none> and password file <none>
I20251214 03:04:15.059464 13842 fs_manager.cc:362] Metadata directory not provided
I20251214 03:04:15.059545 13842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:04:15.059808 13842 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 14 03:04:15 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
I20251214 03:04:15.074215 13842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "283d20db27ff439599884bd12ebb4b7c"
format_stamp: "Formatted at 2025-12-14 03:04:15 on dist-test-slave-k7z7"
server_key: "951ceed350c8a71c0490827f1747aa46"
server_key_iv: "462dc74820ce28265bfe642fad58248a"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:15.075165 13842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "283d20db27ff439599884bd12ebb4b7c"
format_stamp: "Formatted at 2025-12-14 03:04:15 on dist-test-slave-k7z7"
server_key: "951ceed350c8a71c0490827f1747aa46"
server_key_iv: "462dc74820ce28265bfe642fad58248a"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:15.082473 13842 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20251214 03:04:15.101626 13858 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:04:15.104563 13842 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20251214 03:04:15.104712 13842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "283d20db27ff439599884bd12ebb4b7c"
format_stamp: "Formatted at 2025-12-14 03:04:15 on dist-test-slave-k7z7"
server_key: "951ceed350c8a71c0490827f1747aa46"
server_key_iv: "462dc74820ce28265bfe642fad58248a"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:15.104857 13842 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:04:15.125824 13842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:04:15.130095 13842 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:04:15.130311 13842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:04:15.131018 13842 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:04:15.132141 13842 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:04:15.132264 13842 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:04:15.132366 13842 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:04:15.132449 13842 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:04:15.143405 13842 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:41803
I20251214 03:04:15.144708 13842 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:04:15.147789 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 13842
I20251214 03:04:15.147917 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:04:15.148003 13971 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:41803 every 8 connection(s)
I20251214 03:04:15.148198 1655 external_mini_cluster.cc:1468] Setting key bf36c4f97ae28d362ebaa8553d6d806c
WARNING: no policy specified for kudu/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.195@KRBTEST.COM" created.
Dec 14 03:04:15 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:04:15.171509 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.151042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:34979 (local address 127.1.157.254:46221)
1214 03:04:15.151194 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:15.151199 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:15.151928 (+ 729us) server_negotiation.cc:408] Connection header received
1214 03:04:15.153042 (+ 1114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:15.153046 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:15.153102 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:15.153153 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:04:15.161208 (+ 8055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.162051 (+ 843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.163137 (+ 1086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.163397 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.166470 (+ 3073us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:04:15.166501 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:04:15.166505 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:04:15.166540 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:04:15.168389 (+ 1849us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:04:15.169280 (+ 891us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:04:15.169284 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:04:15.169287 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:04:15.169346 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:04:15.169780 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:04:15.169785 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:04:15.169787 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:04:15.169997 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:04:15.170086 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:15.171135 (+ 1049us) server_negotiation.cc:300] Negotiation successful
1214 03:04:15.171321 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251214 03:04:15.173501 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.150037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:46221 (local address 127.1.157.194:34979)
1214 03:04:15.151789 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
1214 03:04:15.151834 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1214 03:04:15.152806 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:04:15.153566 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:04:15.153578 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:04:15.153940 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:04:15.160922 (+ 6982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.160951 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.162277 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.162281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:04:15.162986 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.162999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.163694 (+ 695us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.164444 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:04:15.164483 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:04:15.166241 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:04:15.168702 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:04:15.168715 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:04:15.168731 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:04:15.169098 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:04:15.169478 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:04:15.169481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:04:15.169484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:04:15.169658 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:04:15.170162 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:04:15.170170 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:04:15.170801 (+ 631us) client_negotiation.cc:769] Sending connection context
1214 03:04:15.172045 (+ 1244us) client_negotiation.cc:240] Negotiation successful
1214 03:04:15.172493 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1537,"thread_start_us":134,"threads_started":1}
I20251214 03:04:15.175375 13972 heartbeater.cc:344] Connected to a master server at 127.1.157.254:46221
I20251214 03:04:15.175650 13972 heartbeater.cc:461] Registering TS with master...
I20251214 03:04:15.176329 13972 heartbeater.cc:507] Master 127.1.157.254:46221 requested a full tablet report, sending...
I20251214 03:04:15.177592 13619 ts_manager.cc:194] Registered new tserver with Master: 283d20db27ff439599884bd12ebb4b7c (127.1.157.194:41803)
I20251214 03:04:15.178390 13619 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.194@KRBTEST.COM'} at 127.1.157.194:34979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:04:15.208081 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.157.212:39389/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.195:0
--local_ip_for_outbound_sockets=127.1.157.195
--webserver_interface=127.1.157.195
--webserver_port=0
--tserver_master_addrs=127.1.157.254:46221
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 14 03:04:15 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
W20251214 03:04:15.340627 13979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:04:15.341022 13979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:04:15.341150 13979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:04:15.346030 13979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:04:15.346176 13979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:04:15.346346 13979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.195
I20251214 03:04:15.352599 13979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.157.212:39389/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.157.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.195
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:04:15.354259 13979 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:04:15.355409 13979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:04:15.363504 13984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:04:15.363508 13985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:04:15.364750 13987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:04:15.364852 13979 server_base.cc:1047] running on GCE node
I20251214 03:04:15.365347 13979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:04:15.366019 13979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:04:15.367239 13979 hybrid_clock.cc:648] HybridClock initialized: now 1765681455367189 us; error 69 us; skew 500 ppm
Dec 14 03:04:15 dist-test-slave-k7z7 krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:04:15.370927 13979 init.cc:377] Logged in from keytab as kudu/127.1.157.195@KRBTEST.COM (short username kudu)
I20251214 03:04:15.372467 13979 webserver.cc:492] Webserver started at http://127.1.157.195:42093/ using document root <none> and password file <none>
I20251214 03:04:15.373265 13979 fs_manager.cc:362] Metadata directory not provided
I20251214 03:04:15.373354 13979 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:04:15.373607 13979 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 14 03:04:15 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
I20251214 03:04:15.389518 13979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/data/instance:
uuid: "55419d363a97402abe17cde5860b1f56"
format_stamp: "Formatted at 2025-12-14 03:04:15 on dist-test-slave-k7z7"
server_key: "3b54e15bcf6e850ff4d120c9d3fc8f08"
server_key_iv: "b2a6b2c6b8479e0ec5c5d5d2c0188f01"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:15.390200 13979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance:
uuid: "55419d363a97402abe17cde5860b1f56"
format_stamp: "Formatted at 2025-12-14 03:04:15 on dist-test-slave-k7z7"
server_key: "3b54e15bcf6e850ff4d120c9d3fc8f08"
server_key_iv: "b2a6b2c6b8479e0ec5c5d5d2c0188f01"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:15.394865 13979 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251214 03:04:15.417574 13994 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:04:15.418781 13979 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20251214 03:04:15.418910 13979 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/wal
uuid: "55419d363a97402abe17cde5860b1f56"
format_stamp: "Formatted at 2025-12-14 03:04:15 on dist-test-slave-k7z7"
server_key: "3b54e15bcf6e850ff4d120c9d3fc8f08"
server_key_iv: "b2a6b2c6b8479e0ec5c5d5d2c0188f01"
server_key_version: "kuduclusterkey@0"
I20251214 03:04:15.419024 13979 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:04:15.432197 13979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:04:15.434953 13979 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:04:15.435201 13979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:04:15.435887 13979 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:04:15.436920 13979 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:04:15.436971 13979 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:04:15.437022 13979 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:04:15.437049 13979 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:04:15.469110 13979 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.195:46103
I20251214 03:04:15.472219 13979 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/data/info.pb
I20251214 03:04:15.474031 14107 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.195:46103 every 8 connection(s)
I20251214 03:04:15.478058 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 13979
I20251214 03:04:15.478196 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/wal/instance
I20251214 03:04:15.478484 1655 external_mini_cluster.cc:1468] Setting key 117ecb71e544af25defb0ae3f9d6a522
Dec 14 03:04:15 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.195@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:04:15.494799 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.483085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.195:41383 (local address 127.1.157.254:46221)
1214 03:04:15.483239 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:15.483244 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:15.483263 (+ 19us) server_negotiation.cc:408] Connection header received
1214 03:04:15.483873 (+ 610us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:15.483877 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:15.483936 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:15.484014 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:04:15.485728 (+ 1714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.486563 (+ 835us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.487472 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.487682 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.490630 (+ 2948us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:04:15.490660 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:04:15.490663 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:04:15.490696 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:04:15.492455 (+ 1759us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:04:15.493061 (+ 606us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:04:15.493065 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:04:15.493067 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:04:15.493116 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:04:15.493453 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:04:15.493457 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:04:15.493459 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:04:15.493659 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:04:15.493723 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:15.494450 (+ 727us) server_negotiation.cc:300] Negotiation successful
1214 03:04:15.494614 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251214 03:04:15.496440 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.482030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:46221 (local address 127.1.157.195:41383)
1214 03:04:15.482550 (+ 520us) negotiation.cc:107] Waiting for socket to connect
1214 03:04:15.482595 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1214 03:04:15.483615 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:04:15.484198 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:04:15.484210 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:04:15.484630 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:04:15.485511 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.485532 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.486699 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.486704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:04:15.487320 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.487333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.487955 (+ 622us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.488717 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:04:15.488755 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:04:15.490393 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:04:15.492615 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:04:15.492621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:04:15.492636 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:04:15.492954 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:04:15.493234 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:04:15.493237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:04:15.493239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:04:15.493361 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:04:15.493807 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:04:15.493815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:04:15.494144 (+ 329us) client_negotiation.cc:769] Sending connection context
1214 03:04:15.495267 (+ 1123us) client_negotiation.cc:240] Negotiation successful
1214 03:04:15.495630 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":139,"threads_started":1}
I20251214 03:04:15.499135 14108 heartbeater.cc:344] Connected to a master server at 127.1.157.254:46221
I20251214 03:04:15.499433 14108 heartbeater.cc:461] Registering TS with master...
I20251214 03:04:15.499979 14108 heartbeater.cc:507] Master 127.1.157.254:46221 requested a full tablet report, sending...
I20251214 03:04:15.501226 13619 ts_manager.cc:194] Registered new tserver with Master: 55419d363a97402abe17cde5860b1f56 (127.1.157.195:46103)
I20251214 03:04:15.502142 13619 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.195@KRBTEST.COM'} at 127.1.157.195:41383
I20251214 03:04:15.504848 1655 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 14 03:04:15 dist-test-slave-k7z7 krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
I20251214 03:04:15.517372 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.509889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.1.157.254:46221)
1214 03:04:15.510029 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:15.510032 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:15.510184 (+ 152us) server_negotiation.cc:408] Connection header received
1214 03:04:15.510325 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:15.510327 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:15.510370 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:15.510437 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:04:15.511366 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.511851 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.512448 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.512627 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.513931 (+ 1304us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:04:15.513951 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:04:15.513954 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:04:15.513983 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:04:15.515636 (+ 1653us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:04:15.516095 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:04:15.516099 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:04:15.516101 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:04:15.516151 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:04:15.516405 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:04:15.516408 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:04:15.516410 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:04:15.516605 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:04:15.516708 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:15.517036 (+ 328us) server_negotiation.cc:300] Negotiation successful
1214 03:04:15.517160 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251214 03:04:15.524542 13619 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 03:04:15.528708 13619 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 03:04:15.550611 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.541852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:46103 (local address 127.0.0.1:34550)
1214 03:04:15.545365 (+ 3513us) negotiation.cc:107] Waiting for socket to connect
1214 03:04:15.545378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1214 03:04:15.545449 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:04:15.547717 (+ 2268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:04:15.547721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:04:15.547740 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:04:15.547999 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.548007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.549308 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.549312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:04:15.550196 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.550208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.550315 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.550333 (+ 18us) client_negotiation.cc:769] Sending connection context
1214 03:04:15.550376 (+ 43us) client_negotiation.cc:240] Negotiation successful
1214 03:04:15.550439 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3451,"thread_start_us":85,"threads_started":1}
I20251214 03:04:15.552165 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.539791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.1.157.193:39101)
1214 03:04:15.545713 (+ 5922us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:15.545720 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:15.545746 (+ 26us) server_negotiation.cc:408] Connection header received
1214 03:04:15.545837 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:15.545844 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:15.546065 (+ 221us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:15.546211 (+ 146us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:04:15.546921 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.548126 (+ 1205us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.550531 (+ 2405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.551370 (+ 839us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.551494 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:15.551603 (+ 109us) server_negotiation.cc:300] Negotiation successful
1214 03:04:15.551737 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5789,"thread_start_us":83,"threads_started":1}
I20251214 03:04:15.553959 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.540631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:41803 (local address 127.0.0.1:45554)
1214 03:04:15.545063 (+ 4432us) negotiation.cc:107] Waiting for socket to connect
1214 03:04:15.545075 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:04:15.545151 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:04:15.550925 (+ 5774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:04:15.550929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:04:15.550949 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:04:15.551205 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.551214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.552590 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.552594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:04:15.553534 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.553545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.553666 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.553684 (+ 18us) client_negotiation.cc:769] Sending connection context
1214 03:04:15.553778 (+ 94us) client_negotiation.cc:240] Negotiation successful
1214 03:04:15.553835 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4360,"thread_start_us":101,"threads_started":1}
I20251214 03:04:15.554456 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.539014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:39101 (local address 127.0.0.1:57050)
1214 03:04:15.544623 (+ 5609us) negotiation.cc:107] Waiting for socket to connect
1214 03:04:15.544676 (+ 53us) client_negotiation.cc:174] Beginning negotiation
1214 03:04:15.544828 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:04:15.546414 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:04:15.546421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:04:15.546447 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:04:15.546774 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.546784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.548264 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.548268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:04:15.550063 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.550075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.550182 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.550245 (+ 63us) client_negotiation.cc:769] Sending connection context
1214 03:04:15.550342 (+ 97us) client_negotiation.cc:240] Negotiation successful
1214 03:04:15.550442 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5392,"thread_start_us":115,"threads_started":1}
I20251214 03:04:15.554728 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.540988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.1.157.194:41803)
1214 03:04:15.549067 (+ 8079us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:15.549074 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:15.549100 (+ 26us) server_negotiation.cc:408] Connection header received
1214 03:04:15.549203 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:15.549209 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:15.549388 (+ 179us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:15.549510 (+ 122us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:04:15.551321 (+ 1811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.552348 (+ 1027us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.553696 (+ 1348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.554273 (+ 577us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.554384 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:15.554448 (+ 64us) server_negotiation.cc:300] Negotiation successful
1214 03:04:15.554533 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":97,"threads_started":1}
I20251214 03:04:15.557758 13897 tablet_service.cc:1505] Processing CreateTablet for tablet f846d81a7b2543c6b2e1ff9b066842ad (DEFAULT_TABLE table=test-table [id=87830d88c1d640d69312aa5a0eb2a908]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:04:15.557758 13768 tablet_service.cc:1505] Processing CreateTablet for tablet f846d81a7b2543c6b2e1ff9b066842ad (DEFAULT_TABLE table=test-table [id=87830d88c1d640d69312aa5a0eb2a908]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:04:15.558759 13768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f846d81a7b2543c6b2e1ff9b066842ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:04:15.558759 13897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f846d81a7b2543c6b2e1ff9b066842ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:04:15.566558 14129 tablet_bootstrap.cc:492] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472: Bootstrap starting.
I20251214 03:04:15.568363 14128 tablet_bootstrap.cc:492] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c: Bootstrap starting.
I20251214 03:04:15.568857 14129 tablet_bootstrap.cc:654] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472: Neither blocks nor log segments found. Creating new log.
I20251214 03:04:15.569749 14129 log.cc:826] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472: Log is configured to *not* fsync() on all Append() calls
I20251214 03:04:15.570478 14128 tablet_bootstrap.cc:654] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c: Neither blocks nor log segments found. Creating new log.
I20251214 03:04:15.571637 14128 log.cc:826] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c: Log is configured to *not* fsync() on all Append() calls
I20251214 03:04:15.571751 14129 tablet_bootstrap.cc:492] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472: No bootstrap required, opened a new log
I20251214 03:04:15.571933 14129 ts_tablet_manager.cc:1403] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251214 03:04:15.572901 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.542235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.1.157.195:46103)
1214 03:04:15.547087 (+ 4852us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:15.547094 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:15.547117 (+ 23us) server_negotiation.cc:408] Connection header received
1214 03:04:15.547202 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:15.547207 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:15.547394 (+ 187us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:15.547545 (+ 151us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:04:15.548137 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.549163 (+ 1026us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.551417 (+ 2254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.551952 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.552045 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:15.571483 (+ 19438us) server_negotiation.cc:300] Negotiation successful
1214 03:04:15.571625 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4736,"thread_start_us":135,"threads_started":1}
I20251214 03:04:15.574544 14128 tablet_bootstrap.cc:492] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c: No bootstrap required, opened a new log
I20251214 03:04:15.574751 14128 ts_tablet_manager.cc:1403] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20251214 03:04:15.575634 14129 raft_consensus.cc:359] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 } } peers { permanent_uuid: "283d20db27ff439599884bd12ebb4b7c" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41803 } } peers { permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 } }
I20251214 03:04:15.575843 14129 raft_consensus.cc:385] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:04:15.575893 14129 raft_consensus.cc:740] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723771a6707f49339f986cf02d934472, State: Initialized, Role: FOLLOWER
I20251214 03:04:15.576397 14129 consensus_queue.cc:260] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 } } peers { permanent_uuid: "283d20db27ff439599884bd12ebb4b7c" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41803 } } peers { permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 } }
I20251214 03:04:15.576283 14040 tablet_service.cc:1505] Processing CreateTablet for tablet f846d81a7b2543c6b2e1ff9b066842ad (DEFAULT_TABLE table=test-table [id=87830d88c1d640d69312aa5a0eb2a908]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 03:04:15.577322 14129 ts_tablet_manager.cc:1434] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20251214 03:04:15.577692 14040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f846d81a7b2543c6b2e1ff9b066842ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:04:15.577714 13834 heartbeater.cc:499] Master 127.1.157.254:46221 was elected leader, sending a full tablet report...
I20251214 03:04:15.577849 14128 raft_consensus.cc:359] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 } } peers { permanent_uuid: "283d20db27ff439599884bd12ebb4b7c" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41803 } } peers { permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 } }
I20251214 03:04:15.578058 14128 raft_consensus.cc:385] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:04:15.578114 14128 raft_consensus.cc:740] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283d20db27ff439599884bd12ebb4b7c, State: Initialized, Role: FOLLOWER
I20251214 03:04:15.578581 14128 consensus_queue.cc:260] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 } } peers { permanent_uuid: "283d20db27ff439599884bd12ebb4b7c" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41803 } } peers { permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 } }
I20251214 03:04:15.580435 14128 ts_tablet_manager.cc:1434] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20251214 03:04:15.580821 13972 heartbeater.cc:499] Master 127.1.157.254:46221 was elected leader, sending a full tablet report...
I20251214 03:04:15.585299 14134 tablet_bootstrap.cc:492] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: Bootstrap starting.
I20251214 03:04:15.587590 14134 tablet_bootstrap.cc:654] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: Neither blocks nor log segments found. Creating new log.
I20251214 03:04:15.589082 14134 log.cc:826] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: Log is configured to *not* fsync() on all Append() calls
I20251214 03:04:15.591339 14134 tablet_bootstrap.cc:492] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: No bootstrap required, opened a new log
I20251214 03:04:15.591681 14134 ts_tablet_manager.cc:1403] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251214 03:04:15.594766 14134 raft_consensus.cc:359] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 } } peers { permanent_uuid: "283d20db27ff439599884bd12ebb4b7c" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41803 } } peers { permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 } }
I20251214 03:04:15.595021 14134 raft_consensus.cc:385] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:04:15.595083 14134 raft_consensus.cc:740] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55419d363a97402abe17cde5860b1f56, State: Initialized, Role: FOLLOWER
I20251214 03:04:15.595587 14134 consensus_queue.cc:260] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 } } peers { permanent_uuid: "283d20db27ff439599884bd12ebb4b7c" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41803 } } peers { permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 } }
I20251214 03:04:15.596354 14108 heartbeater.cc:499] Master 127.1.157.254:46221 was elected leader, sending a full tablet report...
I20251214 03:04:15.596580 14134 ts_tablet_manager.cc:1434] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20251214 03:04:15.633409 14133 raft_consensus.cc:493] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:04:15.633622 14133 raft_consensus.cc:515] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 } } peers { permanent_uuid: "283d20db27ff439599884bd12ebb4b7c" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41803 } } peers { permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 } }
I20251214 03:04:15.634958 14133 leader_election.cc:290] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 723771a6707f49339f986cf02d934472 (127.1.157.193:39101), 55419d363a97402abe17cde5860b1f56 (127.1.157.195:46103)
I20251214 03:04:15.639575 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.635494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:39101 (local address 127.1.157.194:57647)
1214 03:04:15.636108 (+ 614us) negotiation.cc:107] Waiting for socket to connect
1214 03:04:15.636123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1214 03:04:15.636223 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:04:15.636942 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:04:15.636946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:04:15.636965 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:04:15.637234 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.637242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.638283 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.638287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:04:15.639175 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.639184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.639288 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.639310 (+ 22us) client_negotiation.cc:769] Sending connection context
1214 03:04:15.639357 (+ 47us) client_negotiation.cc:240] Negotiation successful
1214 03:04:15.639416 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529}
I20251214 03:04:15.640618 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.636483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:57647 (local address 127.1.157.193:39101)
1214 03:04:15.636617 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:15.636621 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:15.636636 (+ 15us) server_negotiation.cc:408] Connection header received
1214 03:04:15.636691 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:15.636694 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:15.636745 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:15.636823 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:04:15.637364 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.638159 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.639883 (+ 1724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.640331 (+ 448us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.640366 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:15.640414 (+ 48us) server_negotiation.cc:300] Negotiation successful
1214 03:04:15.640483 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251214 03:04:15.640681 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.636641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.194:49851 (local address 127.1.157.195:46103)
1214 03:04:15.636764 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:15.636768 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:15.636977 (+ 209us) server_negotiation.cc:408] Connection header received
1214 03:04:15.637138 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:15.637142 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:15.637221 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:15.637268 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1214 03:04:15.637837 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.638555 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.639625 (+ 1070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.640105 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.640146 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:15.640501 (+ 355us) server_negotiation.cc:300] Negotiation successful
1214 03:04:15.640580 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251214 03:04:15.641470 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.635749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.195:46103 (local address 127.1.157.194:49851)
1214 03:04:15.636911 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
1214 03:04:15.636923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1214 03:04:15.637029 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:04:15.637398 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:04:15.637401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:04:15.637419 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1214 03:04:15.637707 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.637713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.638683 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.638687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:04:15.639496 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:04:15.639503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.640349 (+ 846us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.640366 (+ 17us) client_negotiation.cc:769] Sending connection context
1214 03:04:15.641233 (+ 867us) client_negotiation.cc:240] Negotiation successful
1214 03:04:15.641319 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"thread_start_us":74,"threads_started":1}
I20251214 03:04:15.641403 13786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f846d81a7b2543c6b2e1ff9b066842ad" candidate_uuid: "283d20db27ff439599884bd12ebb4b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723771a6707f49339f986cf02d934472" is_pre_election: true
I20251214 03:04:15.641743 13786 raft_consensus.cc:2468] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283d20db27ff439599884bd12ebb4b7c in term 0.
I20251214 03:04:15.642288 14047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f846d81a7b2543c6b2e1ff9b066842ad" candidate_uuid: "283d20db27ff439599884bd12ebb4b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55419d363a97402abe17cde5860b1f56" is_pre_election: true
I20251214 03:04:15.642567 14047 raft_consensus.cc:2468] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283d20db27ff439599884bd12ebb4b7c in term 0.
I20251214 03:04:15.643086 13862 leader_election.cc:304] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283d20db27ff439599884bd12ebb4b7c, 723771a6707f49339f986cf02d934472; no voters:
I20251214 03:04:15.643600 14133 raft_consensus.cc:2804] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 03:04:15.643666 14133 raft_consensus.cc:493] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:04:15.643700 14133 raft_consensus.cc:3060] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:04:15.644888 14133 raft_consensus.cc:515] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 } } peers { permanent_uuid: "283d20db27ff439599884bd12ebb4b7c" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41803 } } peers { permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 } }
I20251214 03:04:15.645319 14133 leader_election.cc:290] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [CANDIDATE]: Term 1 election: Requested vote from peers 723771a6707f49339f986cf02d934472 (127.1.157.193:39101), 55419d363a97402abe17cde5860b1f56 (127.1.157.195:46103)
I20251214 03:04:15.645992 13786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f846d81a7b2543c6b2e1ff9b066842ad" candidate_uuid: "283d20db27ff439599884bd12ebb4b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723771a6707f49339f986cf02d934472"
I20251214 03:04:15.646116 13786 raft_consensus.cc:3060] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:04:15.647373 13786 raft_consensus.cc:2468] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283d20db27ff439599884bd12ebb4b7c in term 1.
I20251214 03:04:15.647954 13862 leader_election.cc:304] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283d20db27ff439599884bd12ebb4b7c, 723771a6707f49339f986cf02d934472; no voters:
I20251214 03:04:15.647977 14047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f846d81a7b2543c6b2e1ff9b066842ad" candidate_uuid: "283d20db27ff439599884bd12ebb4b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55419d363a97402abe17cde5860b1f56"
I20251214 03:04:15.648104 14047 raft_consensus.cc:3060] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:04:15.648195 14133 raft_consensus.cc:2804] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:04:15.648411 14133 raft_consensus.cc:697] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [term 1 LEADER]: Becoming Leader. State: Replica: 283d20db27ff439599884bd12ebb4b7c, State: Running, Role: LEADER
I20251214 03:04:15.648752 14133 consensus_queue.cc:237] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 } } peers { permanent_uuid: "283d20db27ff439599884bd12ebb4b7c" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41803 } } peers { permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 } }
I20251214 03:04:15.649482 14047 raft_consensus.cc:2468] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283d20db27ff439599884bd12ebb4b7c in term 1.
W20251214 03:04:15.651901 13973 tablet.cc:2378] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 03:04:15.652954 13619 catalog_manager.cc:5654] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 283d20db27ff439599884bd12ebb4b7c (127.1.157.194). New cstate: current_term: 1 leader_uuid: "283d20db27ff439599884bd12ebb4b7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "283d20db27ff439599884bd12ebb4b7c" member_type: VOTER last_known_addr { host: "127.1.157.194" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 } health_report { overall_health: UNKNOWN } } }
I20251214 03:04:15.694061 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.682818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.1.157.194:41803)
1214 03:04:15.689390 (+ 6572us) server_negotiation.cc:207] Beginning negotiation
1214 03:04:15.689396 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1214 03:04:15.689417 (+ 21us) server_negotiation.cc:408] Connection header received
1214 03:04:15.689528 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:04:15.689531 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:04:15.689618 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:04:15.689755 (+ 137us) server_negotiation.cc:227] Negotiated authn=TOKEN
1214 03:04:15.690383 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.691133 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:04:15.692367 (+ 1234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:04:15.692570 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:04:15.692690 (+ 120us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1214 03:04:15.693304 (+ 614us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1214 03:04:15.693429 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1214 03:04:15.693861 (+ 432us) server_negotiation.cc:300] Negotiation successful
1214 03:04:15.693922 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6433}
I20251214 03:04:15.708278 13786 raft_consensus.cc:1275] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472 [term 1 FOLLOWER]: Refusing update from remote peer 283d20db27ff439599884bd12ebb4b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:04:15.708279 14047 raft_consensus.cc:1275] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56 [term 1 FOLLOWER]: Refusing update from remote peer 283d20db27ff439599884bd12ebb4b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 03:04:15.709007 14138 consensus_queue.cc:1048] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "55419d363a97402abe17cde5860b1f56" member_type: VOTER last_known_addr { host: "127.1.157.195" port: 46103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:04:15.709254 14133 consensus_queue.cc:1048] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "723771a6707f49339f986cf02d934472" member_type: VOTER last_known_addr { host: "127.1.157.193" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 03:04:15.726401 14141 mvcc.cc:204] Tried to move back new op lower bound from 7232231242570948608 to 7232231242342727680. Current Snapshot: MvccSnapshot[applied={T|T < 7232231242570948608}]
I20251214 03:04:15.728055 14142 mvcc.cc:204] Tried to move back new op lower bound from 7232231242570948608 to 7232231242342727680. Current Snapshot: MvccSnapshot[applied={T|T < 7232231242570948608}]
I20251214 03:04:15.740023 13619 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:04:15.740192 13619 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20251214 03:04:15.742980 13619 catalog_manager.cc:5941] T 00000000000000000000000000000000 P f90e6242c6284c7aae679482a5e9ff1c: Sending DeleteTablet for 3 replicas of tablet f846d81a7b2543c6b2e1ff9b066842ad
I20251214 03:04:15.743902 13897 tablet_service.cc:1552] Processing DeleteTablet for tablet f846d81a7b2543c6b2e1ff9b066842ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:04:15 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:45554
I20251214 03:04:15.744372 13768 tablet_service.cc:1552] Processing DeleteTablet for tablet f846d81a7b2543c6b2e1ff9b066842ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:04:15 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:57050
I20251214 03:04:15.744565 14040 tablet_service.cc:1552] Processing DeleteTablet for tablet f846d81a7b2543c6b2e1ff9b066842ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 03:04:15 UTC) from {username='kudu', principal='kudu/127.1.157.254@KRBTEST.COM'} at 127.0.0.1:34550
I20251214 03:04:15.745502 14151 tablet_replica.cc:333] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472: stopping tablet replica
I20251214 03:04:15.745810 14151 raft_consensus.cc:2243] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:04:15.745914 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 13701
I20251214 03:04:15.746176 14151 raft_consensus.cc:2272] T f846d81a7b2543c6b2e1ff9b066842ad P 723771a6707f49339f986cf02d934472 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 03:04:15.752605 13606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.157.193:39101 (error 108)
I20251214 03:04:15.753414 14150 tablet_replica.cc:333] T f846d81a7b2543c6b2e1ff9b066842ad P 283d20db27ff439599884bd12ebb4b7c: stopping tablet replica
I20251214 03:04:15.753690 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 13842
I20251214 03:04:15.754671 14152 tablet_replica.cc:333] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: stopping tablet replica
I20251214 03:04:15.754930 14152 raft_consensus.cc:2243] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 03:04:15.755312 14152 raft_consensus.cc:2272] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 03:04:15.756879 14152 ts_tablet_manager.cc:1916] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 03:04:15.759531 14152 ts_tablet_manager.cc:1929] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 03:04:15.759632 14152 log.cc:1199] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: Deleting WAL directory at /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765681312865542-1655-0/minicluster-data/ts-2/wal/wals/f846d81a7b2543c6b2e1ff9b066842ad
I20251214 03:04:15.759964 14152 ts_tablet_manager.cc:1950] T f846d81a7b2543c6b2e1ff9b066842ad P 55419d363a97402abe17cde5860b1f56: Deleting consensus metadata
I20251214 03:04:15.761292 13606 catalog_manager.cc:4985] TS 55419d363a97402abe17cde5860b1f56 (127.1.157.195:46103): tablet f846d81a7b2543c6b2e1ff9b066842ad (table test-table [id=87830d88c1d640d69312aa5a0eb2a908]) successfully deleted
I20251214 03:04:15.762256 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 13979
I20251214 03:04:15.763721 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.763423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.194:41803 (local address 127.0.0.1:45566)
1214 03:04:15.763553 (+ 130us) negotiation.cc:107] Waiting for socket to connect
1214 03:04:15.763627 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.194:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20251214 03:04:15.763854 13604 catalog_manager.cc:4712] TS 283d20db27ff439599884bd12ebb4b7c (127.1.157.194:41803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f846d81a7b2543c6b2e1ff9b066842ad: Network error: Client connection negotiation failed: client connection to 127.1.157.194:41803: connect: Connection refused (error 111)
I20251214 03:04:15.764808 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:04:15.764536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.193:39101 (local address 127.0.0.1:57062)
1214 03:04:15.764694 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1214 03:04:15.764740 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.193:39101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20251214 03:04:15.770437 1655 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_d3z9H/build/debug/bin/kudu with pid 13587
I20251214 03:04:15.776149 1655 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20251214 03:04:16.111400 1655 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20251214 03:04:16.111536 1655 mini_ranger.cc:66] Stopping Ranger...
I20251214 03:04:16.447096 1655 mini_ranger.cc:68] Stopped Ranger
2025-12-14 03:04:16.447 UTC [13147] LOG: received smart shutdown request
2025-12-14 03:04:16.451 UTC [13147] LOG: background worker "logical replication launcher" (PID 13155) exited with exit code 1
2025-12-14 03:04:16.451 UTC [13150] LOG: shutting down
2025-12-14 03:04:16.453 UTC [13150] LOG: checkpoint starting: shutdown immediate
2025-12-14 03:04:17.443 UTC [13150] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.038 s, sync=0.946 s, total=0.992 s; sync files=1108, longest=0.007 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045148, redo lsn=0/2045148
2025-12-14 03:04:17.451 UTC [13147] LOG: database system is shut down
2025-12-14T03:04:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47160 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 14 03:04:17 dist-test-slave-k7z7 krb5kdc[14162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 14 03:04:17 dist-test-slave-k7z7 krb5kdc[14162](info): set up 2 sockets
Dec 14 03:04:17 dist-test-slave-k7z7 krb5kdc[14162](info): commencing operation
krb5kdc: starting...
W20251214 03:04:19.385317 1655 mini_kdc.cc:121] Time spent starting KDC: real 1.818s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/test-user.keytab.
Dec 14 03:04:19 dist-test-slave-k7z7 krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-14T03:04:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T03:04:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/rangeradmin_127.1.157.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/rangeradmin_127.1.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/rangerlookup_127.1.157.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/rangerlookup_127.1.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/HTTP_127.1.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/HTTP_127.1.157.212@KRBTEST.COM.keytab.
I20251214 03:04:19.540445 1655 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_d3z9H/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/postgres -l logfile start
2025-12-14 03:04:22.110 UTC [14193] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-12-14 03:04:22.110 UTC [14193] LOG: listening on IPv4 address "127.1.157.212", port 48911
2025-12-14 03:04:22.114 UTC [14193] LOG: listening on Unix socket "/tmp/.s.PGSQL.48911"
2025-12-14 03:04:22.119 UTC [14198] LOG: database system was shut down at 2025-12-14 03:04:20 UTC
2025-12-14 03:04:22.124 UTC [14193] LOG: database system is ready to accept connections
I20251214 03:04:23.885147 1655 mini_postgres.cc:96] Postgres bound to 48911
2025-12-14 03:04:23.890 UTC [14204] FATAL: database "slave" does not exist
127.1.157.212:48911 - accepting connections
I20251214 03:04:23.891202 1655 mini_ranger.cc:161] Starting Ranger...
I20251214 03:04:23.909066 1655 mini_ranger.cc:84] Created miniranger Postgres user
I20251214 03:04:23.975443 1655 mini_ranger.cc:87] Created ranger Postgres database
I20251214 03:04:23.975530 1655 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-admin
2025-12-14 03:04:24,246 [I] DB FLAVOR :POSTGRES
2025-12-14 03:04:24,247 [I] --------- Verifying Ranger DB connection ---------
2025-12-14 03:04:24,247 [I] Checking connection..
2025-12-14 03:04:24,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-12-14 03:04:24,572 [I] Checking connection passed.
2025-12-14 03:04:24,572 [I] --------- Verifying version history table ---------
2025-12-14 03:04:24,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-14 03:04:24,889 [I] Table x_db_version_h does not exist in database ranger
2025-12-14 03:04:24,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-14 03:04:25,209 [I] Table x_db_version_h does not exist in database ranger
2025-12-14 03:04:25,210 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-12-14 03:04:25,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-12-14 03:04:25.526 UTC [14271] WARNING: there is no transaction in progress
2025-12-14 03:04:25,532 [I] create_dbversion_catalog.sql file imported successfully
2025-12-14 03:04:25,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-14 03:04:25,860 [I] Table x_db_version_h already exists in database 'ranger'
2025-12-14 03:04:25,860 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-14 03:04:25,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-12-14 03:04:26,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-12-14 03:04:26,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k7z7.c.gcp-upstream.internal','N') ;"
2025-12-14 03:04:26,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-14 03:04:27,158 [I] Table x_portal_user does not exist in database ranger
2025-12-14 03:04:27,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-14 03:04:27,477 [I] Table x_policy_ref_group does not exist in database ranger
2025-12-14 03:04:27,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-14 03:04:27,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-14 03:04:28,102 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-12-14 03:04:28,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-12-14 03:04:28.957 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:28.972 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:28.988 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.005 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.028 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.041 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.089 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.094 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.103 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.115 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.127 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.136 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.146 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.150 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.154 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29.157 UTC [14388] WARNING: there is no transaction in progress
2025-12-14 03:04:29,626 [I] ranger_core_db_postgres.sql file imported successfully
2025-12-14 03:04:29,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-12-14 03:04:29,952 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-12-14 03:04:29,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-14 03:04:30,260 [I] Table x_portal_user already exists in database 'ranger'
2025-12-14 03:04:30,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-14 03:04:30,576 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-12-14 03:04:30,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-14 03:04:30,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-14 03:04:31,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k7z7.c.gcp-upstream.internal';"
2025-12-14 03:04:31,511 [I] CORE_DB_SCHEMA import status has been updated
2025-12-14 03:04:31,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-12-14 03:04:31,834 [I] DB_PATCHES have already been applied
I20251214 03:04:31.840193 1655 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-admin:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_d3z9H/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251214 03:04:31.840255 1655 mini_ranger.cc:193] Using host: 127.1.157.212
I20251214 03:04:31.843006 1655 mini_ranger.cc:227] Ranger admin URL: http://127.1.157.212:50545
Dec 14, 2025 3:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 14, 2025 3:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 14, 2025 3:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-admin/ews/webapp
Dec 14, 2025 3:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Dec 14, 2025 3:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-admin/ews/webapp] .....
Dec 14, 2025 3:04:32 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Dec 14, 2025 3:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-admin/ews/webapp].
Dec 14, 2025 3:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Dec 14, 2025 3:04:32 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-50545"]
Dec 14, 2025 3:04:32 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Dec 14, 2025 3:04:32 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Dec 14, 2025 3:04:32 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Dec 14, 2025 3:04:33 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Dec 14, 2025 3:04:33 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Dec 14, 2025 3:04:33 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Dec 14, 2025 3:04:33 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 14, 2025 3:04:33 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
I20251214 03:04:33.846799 1655 mini_ranger.cc:160] Time spent starting Ranger: real 9.956s user 0.005s sys 0.004s
Dec 14, 2025 3:04:33 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Dec 14, 2025 3:04:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Dec 14, 2025 3:04:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
Dec 14, 2025 3:04:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Dec 14, 2025 3:04:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Dec 14, 2025 3:04:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Dec 14, 2025 3:04:53 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-50545"]
I20251214 03:04:54.132457 1655 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/rangerkms_127.1.157.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/rangerkms_127.1.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.157.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.157.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/HTTP_127.1.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.157.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/HTTP_127.1.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Dec 14 03:04:54 dist-test-slave-k7z7 krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681494, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251214 03:04:54.219530 1655 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251214 03:04:54.228302 1655 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251214 03:04:54.326973 1655 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251214 03:04:54.327071 1655 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-kms
I20251214 03:04:54.327088 1655 mini_ranger_kms.cc:227] Using postgres at 127.1.157.212:48911
2025-12-14 03:04:54,485 [I] DB FLAVOR :POSTGRES
2025-12-14 03:04:54,485 [I] --------- Verifying Ranger DB connection ---------
2025-12-14 03:04:54,485 [I] Checking connection
2025-12-14 03:04:54,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-12-14 03:04:54,797 [I] connection success
2025-12-14 03:04:54,797 [I] --------- Verifying Ranger DB tables ---------
2025-12-14 03:04:54,797 [I] Verifying table ranger_masterkey in database rangerkms
2025-12-14 03:04:54,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-12-14 03:04:55,120 [I] Table ranger_masterkey does not exist in database rangerkms
2025-12-14 03:04:55,120 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-14 03:04:55,120 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-12-14 03:04:55,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.157.212:48911/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-12-14 03:04:55,434 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251214 03:04:55.929136 1655 mini_ranger_kms.cc:310] Created kms service
I20251214 03:04:56.149533 1655 mini_ranger_kms.cc:326] Created kudu user
I20251214 03:04:56.191068 1655 mini_ranger_kms.cc:343] Created rangerkms user
Dec 14 03:04:56 dist-test-slave-k7z7 krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681494, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
I20251214 03:04:56.584676 1655 mini_ranger_kms.cc:384] Added ranger policy
I20251214 03:04:56.584898 1655 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-kms:/tmp/dist-test-task_d3z9H/build/debug/bin/postgresql.jar:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_d3z9H/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251214 03:04:56.584919 1655 mini_ranger_kms.cc:240] Using host: 127.1.157.212
I20251214 03:04:56.588243 1655 mini_ranger_kms.cc:276] Ranger KMS PID: 14580
I20251214 03:04:56.588346 1655 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.157.212:38489
Dec 14, 2025 3:04:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 14, 2025 3:04:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 14, 2025 3:04:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Dec 14, 2025 3:04:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-kms/ews/webapp] .....
Dec 14, 2025 3:04:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-kms/ews/webapp].
Dec 14, 2025 3:04:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Dec 14, 2025 3:04:58 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-38489"]
Dec 14, 2025 3:04:58 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Dec 14, 2025 3:04:58 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Dec 14, 2025 3:04:58 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20251214 03:04:58.595126 1655 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.376s user 0.012s sys 0.130s
I20251214 03:04:58.595232 1655 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251214 03:04:58.595309 1655 mini_ranger_kms.cc:401] 127.1.157.212:38489/kms/v1/keys
Dec 14, 2025 3:04:58 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 14 03:04:58 dist-test-slave-k7z7 krb5kdc[14162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1765681498, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.157.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Dec 14 03:04:59 dist-test-slave-k7z7 krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681498, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.157.212@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
Dec 14 03:04:59 dist-test-slave-k7z7 krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681498, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.157.212@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
Dec 14 03:05:01 dist-test-slave-k7z7 krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:05:01.781594 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:41935
--webserver_interface=127.1.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.157.254:41935
--ranger_config_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.157.212:38489/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:05:01.888809 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:05:01.889042 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:05:01.889101 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:05:01.892397 14633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 03:05:01.892465 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:05:01.892488 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:05:01.892503 14633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 03:05:01.892524 14633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 03:05:01.896834 14633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.157.212:38489/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.157.254:41935
--ranger_config_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.157.254:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.157.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:05:01.897814 14633 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:05:01.898643 14633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:05:01.904314 14639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:05:01.904451 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:05:01.904368 14638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:05:01.904345 14633 server_base.cc:1047] running on GCE node
I20251214 03:05:01.905022 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:05:01.905866 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:05:01.907011 14633 hybrid_clock.cc:648] HybridClock initialized: now 1765681501907003 us; error 26 us; skew 500 ppm
Dec 14 03:05:01 dist-test-slave-k7z7 krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681501, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:05:01.909276 14633 init.cc:377] Logged in from keytab as kudu/127.1.157.254@KRBTEST.COM (short username kudu)
I20251214 03:05:01.910194 14633 webserver.cc:492] Webserver started at http://127.1.157.254:42997/ using document root <none> and password file <none>
I20251214 03:05:01.910665 14633 fs_manager.cc:362] Metadata directory not provided
I20251214 03:05:01.910727 14633 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:05:01.910884 14633 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 14 03:05:01 dist-test-slave-k7z7 krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681501, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.254@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
I20251214 03:05:01.965886 14633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/data/instance:
uuid: "3bb33b74e3fd4f6280f535982a747ed0"
format_stamp: "Formatted at 2025-12-14 03:05:01 on dist-test-slave-k7z7"
server_key: "ce95660ad0d0446f69ca6164f8b76426"
server_key_iv: "f5026136ecf5258ece711e788f862907"
server_key_version: "kuduclusterkey@0"
I20251214 03:05:01.966528 14633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/wal/instance:
uuid: "3bb33b74e3fd4f6280f535982a747ed0"
format_stamp: "Formatted at 2025-12-14 03:05:01 on dist-test-slave-k7z7"
server_key: "ce95660ad0d0446f69ca6164f8b76426"
server_key_iv: "f5026136ecf5258ece711e788f862907"
server_key_version: "kuduclusterkey@0"
I20251214 03:05:01.970847 14633 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20251214 03:05:01.988672 14648 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:05:01.990054 14633 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251214 03:05:01.990193 14633 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/wal
uuid: "3bb33b74e3fd4f6280f535982a747ed0"
format_stamp: "Formatted at 2025-12-14 03:05:01 on dist-test-slave-k7z7"
server_key: "ce95660ad0d0446f69ca6164f8b76426"
server_key_iv: "f5026136ecf5258ece711e788f862907"
server_key_version: "kuduclusterkey@0"
I20251214 03:05:01.990336 14633 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:05:02.000360 14633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:05:02.003126 14633 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:05:02.003286 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:05:02.010882 14633 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.254:41935
I20251214 03:05:02.010890 14700 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.254:41935 every 8 connection(s)
I20251214 03:05:02.011883 14633 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/data/info.pb
I20251214 03:05:02.014556 14701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 03:05:02.018047 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 14633
I20251214 03:05:02.018199 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/wal/instance
I20251214 03:05:02.018460 1655 external_mini_cluster.cc:1468] Setting key e4bf4c20fafa6e4543e04b4ed29d4e0c
I20251214 03:05:02.019942 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0: Bootstrap starting.
I20251214 03:05:02.022639 14701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0: Neither blocks nor log segments found. Creating new log.
I20251214 03:05:02.023523 14701 log.cc:826] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0: Log is configured to *not* fsync() on all Append() calls
I20251214 03:05:02.025779 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0: No bootstrap required, opened a new log
Dec 14 03:05:02 dist-test-slave-k7z7 krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
I20251214 03:05:02.028861 14701 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb33b74e3fd4f6280f535982a747ed0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 41935 } }
I20251214 03:05:02.029114 14701 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 03:05:02.029204 14701 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb33b74e3fd4f6280f535982a747ed0, State: Initialized, Role: FOLLOWER
I20251214 03:05:02.029732 14701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb33b74e3fd4f6280f535982a747ed0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 41935 } }
I20251214 03:05:02.029902 14701 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 03:05:02.030007 14701 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 03:05:02.030107 14701 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [term 0 FOLLOWER]: Advancing to term 1
I20251214 03:05:02.031139 14701 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb33b74e3fd4f6280f535982a747ed0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 41935 } }
I20251214 03:05:02.031562 14701 leader_election.cc:304] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bb33b74e3fd4f6280f535982a747ed0; no voters:
I20251214 03:05:02.031919 14701 leader_election.cc:290] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 03:05:02.032030 14706 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 03:05:02.032289 14706 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [term 1 LEADER]: Becoming Leader. State: Replica: 3bb33b74e3fd4f6280f535982a747ed0, State: Running, Role: LEADER
I20251214 03:05:02.032665 14706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb33b74e3fd4f6280f535982a747ed0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 41935 } }
I20251214 03:05:02.033267 14701 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 03:05:02.033979 14706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bb33b74e3fd4f6280f535982a747ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb33b74e3fd4f6280f535982a747ed0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 41935 } } }
I20251214 03:05:02.034159 14706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [sys.catalog]: This master's current role is: LEADER
I20251214 03:05:02.034197 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:02.019868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.1.157.254:41935)
1214 03:05:02.020312 (+ 444us) server_negotiation.cc:207] Beginning negotiation
1214 03:05:02.020324 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1214 03:05:02.020363 (+ 39us) server_negotiation.cc:408] Connection header received
1214 03:05:02.021078 (+ 715us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:05:02.021106 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:05:02.021531 (+ 425us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:05:02.021901 (+ 370us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:05:02.022948 (+ 1047us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:05:02.024015 (+ 1067us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:05:02.024780 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:05:02.025073 (+ 293us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:05:02.027847 (+ 2774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:05:02.027872 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:05:02.027888 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:05:02.027934 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:05:02.030730 (+ 2796us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:05:02.031262 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:05:02.031269 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:05:02.031275 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:05:02.031364 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:05:02.031604 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:05:02.031608 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:05:02.031610 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:05:02.032099 (+ 489us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:05:02.032984 (+ 885us) server_negotiation.cc:1036] Waiting for connection context
1214 03:05:02.033152 (+ 168us) server_negotiation.cc:300] Negotiation successful
1214 03:05:02.033419 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":146,"threads_started":1}
I20251214 03:05:02.034406 14706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bb33b74e3fd4f6280f535982a747ed0. Latest consensus state: current_term: 1 leader_uuid: "3bb33b74e3fd4f6280f535982a747ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb33b74e3fd4f6280f535982a747ed0" member_type: VOTER last_known_addr { host: "127.1.157.254" port: 41935 } } }
I20251214 03:05:02.034607 14706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0 [sys.catalog]: This master's current role is: LEADER
I20251214 03:05:02.034662 14715 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 03:05:02.037760 14715 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 03:05:02.038131 14701 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251214 03:05:02.047573 14715 catalog_manager.cc:1357] Generated new cluster ID: e6e55dafd4314b95b26cb05efdd97283
I20251214 03:05:02.047662 14715 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 03:05:02.069532 14715 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 03:05:02.070947 14715 catalog_manager.cc:1514] Loading token signing keys...
I20251214 03:05:02.083227 14715 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3bb33b74e3fd4f6280f535982a747ed0: Generated new TSK 0
I20251214 03:05:02.084235 14715 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
I20251214 03:05:02.840117 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.157.212:38489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.193:0
--local_ip_for_outbound_sockets=127.1.157.193
--webserver_interface=127.1.157.193
--webserver_port=0
--tserver_master_addrs=127.1.157.254:41935
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251214 03:05:02.963238 14747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:05:02.963485 14747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:05:02.963635 14747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:05:02.967232 14747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:05:02.967303 14747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:05:02.967404 14747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.193
I20251214 03:05:02.972275 14747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.157.212:38489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.157.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.193
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:05:02.973450 14747 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:05:02.974453 14747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:05:02.981446 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:05:02.981514 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:05:02.981781 14747 server_base.cc:1047] running on GCE node
W20251214 03:05:02.981794 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:05:02.982213 14747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:05:02.982801 14747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:05:02.983965 14747 hybrid_clock.cc:648] HybridClock initialized: now 1765681502983950 us; error 34 us; skew 500 ppm
Dec 14 03:05:02 dist-test-slave-k7z7 krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681502, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:05:02.986565 14747 init.cc:377] Logged in from keytab as kudu/127.1.157.193@KRBTEST.COM (short username kudu)
I20251214 03:05:02.987725 14747 webserver.cc:492] Webserver started at http://127.1.157.193:43739/ using document root <none> and password file <none>
I20251214 03:05:02.988287 14747 fs_manager.cc:362] Metadata directory not provided
I20251214 03:05:02.988336 14747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:05:02.988521 14747 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 14 03:05:02 dist-test-slave-k7z7 krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681502, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
I20251214 03:05:03.006364 14747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/data/instance:
uuid: "e5e8593ac391403ab16b8e8b44062110"
format_stamp: "Formatted at 2025-12-14 03:05:02 on dist-test-slave-k7z7"
server_key: "89bd7310b06d8db22a1248c72e82885c"
server_key_iv: "74d8bcba5a93b95be9ec2108af1043de"
server_key_version: "kuduclusterkey@0"
I20251214 03:05:03.007220 14747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance:
uuid: "e5e8593ac391403ab16b8e8b44062110"
format_stamp: "Formatted at 2025-12-14 03:05:02 on dist-test-slave-k7z7"
server_key: "89bd7310b06d8db22a1248c72e82885c"
server_key_iv: "74d8bcba5a93b95be9ec2108af1043de"
server_key_version: "kuduclusterkey@0"
I20251214 03:05:03.026090 14747 fs_manager.cc:696] Time spent creating directory manager: real 0.018s user 0.004s sys 0.000s
I20251214 03:05:03.049504 14764 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:05:03.050918 14747 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.001s sys 0.001s
I20251214 03:05:03.051043 14747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/wal
uuid: "e5e8593ac391403ab16b8e8b44062110"
format_stamp: "Formatted at 2025-12-14 03:05:02 on dist-test-slave-k7z7"
server_key: "89bd7310b06d8db22a1248c72e82885c"
server_key_iv: "74d8bcba5a93b95be9ec2108af1043de"
server_key_version: "kuduclusterkey@0"
I20251214 03:05:03.051157 14747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:05:03.072129 14747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:05:03.078778 14747 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:05:03.079044 14747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:05:03.079774 14747 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:05:03.080734 14747 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:05:03.080830 14747 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:05:03.080935 14747 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:05:03.080972 14747 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:05:03.093645 14747 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.193:40551
I20251214 03:05:03.093703 14879 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.193:40551 every 8 connection(s)
I20251214 03:05:03.094897 14747 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/data/info.pb
I20251214 03:05:03.103053 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 14747
I20251214 03:05:03.103211 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-0/wal/instance
I20251214 03:05:03.103519 1655 external_mini_cluster.cc:1468] Setting key a397593a9a47a798003862ed04a8a276
Dec 14 03:05:03 dist-test-slave-k7z7 krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681502, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.193@KRBTEST.COM for kudu/127.1.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.157.194@KRBTEST.COM; defaulting to no policy
I20251214 03:05:03.116309 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:03.104661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.157.193:53633 (local address 127.1.157.254:41935)
1214 03:05:03.104890 (+ 229us) server_negotiation.cc:207] Beginning negotiation
1214 03:05:03.104895 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1214 03:05:03.105590 (+ 695us) server_negotiation.cc:408] Connection header received
1214 03:05:03.106422 (+ 832us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1214 03:05:03.106425 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1214 03:05:03.106477 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1214 03:05:03.106551 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1214 03:05:03.107843 (+ 1292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:05:03.108563 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1214 03:05:03.109408 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1214 03:05:03.109600 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:05:03.112406 (+ 2806us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1214 03:05:03.112426 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1214 03:05:03.112430 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1214 03:05:03.112458 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1214 03:05:03.114278 (+ 1820us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:05:03.114972 (+ 694us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:05:03.114976 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:05:03.114978 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:05:03.115024 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1214 03:05:03.115293 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1214 03:05:03.115297 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1214 03:05:03.115299 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1214 03:05:03.115443 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1214 03:05:03.115526 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1214 03:05:03.116022 (+ 496us) server_negotiation.cc:300] Negotiation successful
1214 03:05:03.116140 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":75,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.157.194@KRBTEST.COM" created.
I20251214 03:05:03.117688 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:03.103773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.193:53633)
1214 03:05:03.105473 (+ 1700us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:03.105518 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1214 03:05:03.106238 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:05:03.106719 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1214 03:05:03.106727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1214 03:05:03.106985 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
1214 03:05:03.107678 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:05:03.107694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:05:03.108714 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1214 03:05:03.108718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1214 03:05:03.109280 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1214 03:05:03.109289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1214 03:05:03.109851 (+ 562us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1214 03:05:03.110558 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1214 03:05:03.110593 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1214 03:05:03.112230 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1214 03:05:03.114386 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:05:03.114391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:05:03.114400 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:05:03.114700 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:05:03.115105 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1214 03:05:03.115109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1214 03:05:03.115111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1214 03:05:03.115214 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1214 03:05:03.115537 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1214 03:05:03.115544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1214 03:05:03.115784 (+ 240us) client_negotiation.cc:769] Sending connection context
1214 03:05:03.115976 (+ 192us) client_negotiation.cc:240] Negotiation successful
1214 03:05:03.116208 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1497,"thread_start_us":137,"threads_started":1}
I20251214 03:05:03.119176 14880 heartbeater.cc:344] Connected to a master server at 127.1.157.254:41935
I20251214 03:05:03.119423 14880 heartbeater.cc:461] Registering TS with master...
I20251214 03:05:03.120004 14880 heartbeater.cc:507] Master 127.1.157.254:41935 requested a full tablet report, sending...
I20251214 03:05:03.121505 14665 ts_manager.cc:194] Registered new tserver with Master: e5e8593ac391403ab16b8e8b44062110 (127.1.157.193:40551)
I20251214 03:05:03.122601 14665 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.157.193@KRBTEST.COM'} at 127.1.157.193:53633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.1.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.1.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
I20251214 03:05:03.156157 1655 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_d3z9H/build/debug/bin/kudu
/tmp/dist-test-task_d3z9H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.157.212:38489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.157.194:0
--local_ip_for_outbound_sockets=127.1.157.194
--webserver_interface=127.1.157.194
--webserver_port=0
--tserver_master_addrs=127.1.157.254:41935
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.1.157.212:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.1.157.254 from keytab /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20251214 03:05:03.173950 14739 server.cc:273] Received an EOF from the subprocess
F20251214 03:05:03.175124 14725 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f1844851dcd google::LogMessage::Fail() at ??:0
@ 0x7f1844855b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f18448517cc google::LogMessage::Flush() at ??:0
@ 0x7f1844852f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f18465082d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f18465050d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f1846509005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56064391aa0a std::function<>::operator()() at ??:0
@ 0x7f1845cab7dc kudu::Thread::SuperviseThread() at ??:0
@ 0x7f18465446db start_thread at ??:0
@ 0x7f1843a0871f clone at ??:0
W20251214 03:05:03.279603 14888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 03:05:03.279820 14888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 03:05:03.279877 14888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 03:05:03.283120 14888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251214 03:05:03.283174 14888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 03:05:03.283259 14888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.157.194
I20251214 03:05:03.287370 14888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.157.212:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.157.212:38489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/krb5kdc/kudu.keytab
--principal=kudu/127.1.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.157.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.1.157.254:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.157.194
--log_dir=/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 14 Dec 2025 02:43:16 UTC on 24a791456cd2
build id 9502
I20251214 03:05:03.288406 14888 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 03:05:03.289244 14888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 03:05:03.295557 14908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:05:03.295578 14906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 03:05:03.295579 14905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 03:05:03.296363 14888 server_base.cc:1047] running on GCE node
I20251214 03:05:03.296727 14888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 03:05:03.297273 14888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 03:05:03.298435 14888 hybrid_clock.cc:648] HybridClock initialized: now 1765681503298418 us; error 35 us; skew 500 ppm
Dec 14 03:05:03 dist-test-slave-k7z7 krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681503, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251214 03:05:03.300908 14888 init.cc:377] Logged in from keytab as kudu/127.1.157.194@KRBTEST.COM (short username kudu)
I20251214 03:05:03.301985 14888 webserver.cc:492] Webserver started at http://127.1.157.194:43653/ using document root <none> and password file <none>
I20251214 03:05:03.302538 14888 fs_manager.cc:362] Metadata directory not provided
I20251214 03:05:03.302588 14888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 03:05:03.302762 14888 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 14 03:05:03 dist-test-slave-k7z7 krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765681503, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.157.194@KRBTEST.COM for HTTP/127.1.157.212@KRBTEST.COM
I20251214 03:05:03.318419 14888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/data/instance:
uuid: "ad2b95dd59de4fc7aec94c48a4e1ce2c"
format_stamp: "Formatted at 2025-12-14 03:05:03 on dist-test-slave-k7z7"
server_key: "1f4d9c7a283f68a71f5b5cba208c277d"
server_key_iv: "bed0a39b1adbcf488fc795ded3c48cc6"
server_key_version: "kuduclusterkey@0"
I20251214 03:05:03.319193 14888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance:
uuid: "ad2b95dd59de4fc7aec94c48a4e1ce2c"
format_stamp: "Formatted at 2025-12-14 03:05:03 on dist-test-slave-k7z7"
server_key: "1f4d9c7a283f68a71f5b5cba208c277d"
server_key_iv: "bed0a39b1adbcf488fc795ded3c48cc6"
server_key_version: "kuduclusterkey@0"
I20251214 03:05:03.323911 14888 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251214 03:05:03.339692 14915 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 03:05:03.341225 14888 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251214 03:05:03.341390 14888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/data,/tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/wal
uuid: "ad2b95dd59de4fc7aec94c48a4e1ce2c"
format_stamp: "Formatted at 2025-12-14 03:05:03 on dist-test-slave-k7z7"
server_key: "1f4d9c7a283f68a71f5b5cba208c277d"
server_key_iv: "bed0a39b1adbcf488fc795ded3c48cc6"
server_key_version: "kuduclusterkey@0"
I20251214 03:05:03.341537 14888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 03:05:03.369199 14888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 03:05:03.372905 14888 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 03:05:03.373135 14888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 03:05:03.373907 14888 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 03:05:03.375151 14888 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 03:05:03.375227 14888 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 03:05:03.375293 14888 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 03:05:03.375330 14888 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 03:05:03.386576 14888 rpc_server.cc:307] RPC server started. Bound to: 127.1.157.194:40189
I20251214 03:05:03.386837 15028 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.157.194:40189 every 8 connection(s)
I20251214 03:05:03.387605 14888 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/data/info.pb
I20251214 03:05:03.393494 1655 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_d3z9H/build/debug/bin/kudu as pid 14888
I20251214 03:05:03.393633 1655 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_d3z9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765681312865542-1655-0/minicluster-data/ts-1/wal/instance
I20251214 03:05:03.393920 1655 external_mini_cluster.cc:1468] Setting key 3567b6500215428d357176900aa60d57
W20251214 03:05:03.412325 14160 connection.cc:537] client connection to 127.1.157.254:41935 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251214 03:05:03.413218 15031 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 03:05:03.389831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.194:53669)
1214 03:05:03.390332 (+ 501us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:03.390374 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1214 03:05:03.391100 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 03:05:03.412461 (+ 21361us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":128,"threads_started":1}
W20251214 03:05:03.413681 15029 heartbeater.cc:646] Failed to heartbeat to 127.1.157.254:41935 (0 consecutive failures): Network error: Failed to ping master at 127.1.157.254:41935: Client connection negotiation failed: client connection to 127.1.157.254:41935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251214 03:05:03.414346 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:03.414117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.194:34507)
1214 03:05:03.414233 (+ 116us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:03.414287 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20251214 03:05:03.414891 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:03.414716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.194:36391)
1214 03:05:03.414805 (+ 89us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:03.414842 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20251214 03:05:03.415040 15029 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251214 03:05:04.125360 14880 heartbeater.cc:499] Master 127.1.157.254:41935 was elected leader, sending a full tablet report...
I20251214 03:05:04.126374 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:04.125930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.193:42983)
1214 03:05:04.126190 (+ 260us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:04.126262 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
W20251214 03:05:04.126632 14880 heartbeater.cc:646] Failed to heartbeat to 127.1.157.254:41935 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
I20251214 03:05:04.416253 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:04.415762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.194:47073)
1214 03:05:04.416077 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:04.416140 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20251214 03:05:05.128121 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:05.127587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.193:51203)
1214 03:05:05.127937 (+ 350us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:05.128014 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":93,"threads_started":1}
I20251214 03:05:05.417510 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:05.417014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.194:50109)
1214 03:05:05.417325 (+ 311us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:05.417397 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":90,"threads_started":1}
I20251214 03:05:06.129489 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:06.128997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.193:35531)
1214 03:05:06.129295 (+ 298us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:06.129359 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20251214 03:05:06.418607 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:06.418214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.194:40543)
1214 03:05:06.418455 (+ 241us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:06.418511 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20251214 03:05:07.130911 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:07.130425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.193:38329)
1214 03:05:07.130736 (+ 311us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:07.130802 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
I20251214 03:05:07.419839 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:07.419352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.194:47601)
1214 03:05:07.419653 (+ 301us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:07.419719 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
I20251214 03:05:08.132356 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:08.131876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.193:44875)
1214 03:05:08.132187 (+ 311us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:08.132253 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
I20251214 03:05:08.421308 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1214 03:05:08.420702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.157.254:41935 (local address 127.1.157.194:59647)
1214 03:05:08.421088 (+ 386us) negotiation.cc:107] Waiting for socket to connect
1214 03:05:08.421153 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.157.254:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":94,"threads_started":1}
F20251214 03:05:08.457397 1655 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1765681508 (unix time) try "date -d @1765681508" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000677) received by PID 1655 (TID 0x7f87973c7d80) from PID 1655; stack trace: ***
@ 0x7f87a306d980 (unknown) at ??:0
@ 0x7f87a06a4fb7 gsignal at ??:0
@ 0x7f87a06a6921 abort at ??:0
@ 0x7f87a1912dcd google::LogMessage::Fail() at ??:0
@ 0x7f87a1916b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f87a19127cc google::LogMessage::Flush() at ??:0
@ 0x7f87a1913f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f87a469ecab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f87a469b9c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55c0519c792f kudu::SecurityITest::StartCluster() at ??:0
@ 0x55c0519afc15 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7f87a29220ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f87a2916bdb testing::Test::Run() at ??:0
@ 0x7f87a2916d9d testing::TestInfo::Run() at ??:0
@ 0x7f87a2917377 testing::TestSuite::Run() at ??:0
@ 0x7f87a291777c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f87a292260d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f87a2916e63 testing::UnitTest::Run() at ??:0
@ 0x55c0519ffc7e RUN_ALL_TESTS() at ??:0
@ 0x55c0519fee6b main at ??:0
@ 0x7f87a0687bf7 __libc_start_main at ??:0
@ 0x55c05199706a _start at ??:0
2025-12-14 03:05:08.651 UTC [14505] FATAL: terminating connection due to unexpected postmaster exit
2025-12-14 03:05:08.651 UTC [14611] FATAL: terminating connection due to unexpected postmaster exit
2025-12-14 03:05:08.653 UTC [14503] FATAL: terminating connection due to unexpected postmaster exit
2025-12-14 03:05:08.653 UTC [14504] FATAL: terminating connection due to unexpected postmaster exit
2025-12-14 03:05:08.653 UTC [14507] FATAL: terminating connection due to unexpected postmaster exit
2025-12-14 03:05:08.654 UTC [14508] FATAL: terminating connection due to unexpected postmaster exit